Virtual sharing of physical notebook

ABSTRACT

Methods, systems, apparatuses, and computer-readable media are provided for enabling collaboration between physical writers and virtual writers. In one implementation, the computer-readable medium includes instructions to cause a processor to receive, from an image sensor, image data representing a hand of a first physical writer holding a physical marking implement and engaging with a physical surface to create tangible markings. Information based on the image data is transmitted to a computing device associated with a second virtual writer, to thereby enable the second virtual writer to view the tangible markings created by the first physical writer. Annotation data are received from the computing device, and represent additional markings in relative locations with respect to the tangible markings created by the first physical writer. In response to receiving the annotation data, a wearable extended reality appliance overlays the physical surface with virtual markings in the relative locations.

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of priority of U.S. ProvisionalPatent Application No. 63/226,180, filed on Jul. 28, 2021, U.S.Provisional Patent Application No. 63/299,188, filed on Jan. 13, 2022.U.S. Provisional Patent Application No. 63/307,207, filed on Feb. 7,2022, U.S. Provisional Patent Application No. 63/307,203, filed on Feb.7, 2022, U.S. Provisional Patent Application No. 63/307,217, filed onFeb. 7, 2022, U.S. Provisional Patent Application No. 63/319,423, filedon Mar. 14, 2022, and U.S. Provisional Patent Application No.63/344,727, filed on May 23, 2022, all of which are incorporated hereinby reference in their entirety.

BACKGROUND I. Technical Field

The present disclosure generally relates to the field of extendedreality. More specifically, the present disclosure relates to systems,methods, and devices for providing productivity applications using anextended reality environment.

II. Background Information

For many years, PC users were faced with a productivity dilemma; eitherto limit their mobility (when selecting a desktop computer) or to limittheir screen size (when selecting a laptop computer). One partialsolution to this dilemma is using a docking station. A docking stationis an interface device for connecting a laptop computer with otherdevices. By plugging the laptop computer into the docking station,laptop users can enjoy the increased visibility provided by a largermonitor. But because the large monitor is stationary, the mobility ofthe user—while improved—is still limited. For example, even laptop userswith docking stations do not have the freedom of using two 32″ screensanywhere they want.

Some of the disclosed embodiments are directed to providing a newapproach for solving the productivity dilemma, one that uses extendedreality (XR) to provide a mobile environment that enables users toexperience the comfort of a stationary workspace anywhere they want byproviding virtual desktop-like screens.

SUMMARY

Embodiments consistent with the present disclosure provide systems,methods, and devices for providing and supporting productivityapplications using an extended reality environment.

Some disclosed embodiments may include systems, methods, andnon-transitory computer readable media for performing duty cycle controloperations for wearable extended reality appliances. These embodimentsmay involve receiving data representing virtual content in an extendedreality environment associated with a wearable extended realityappliance; identifying in the extended reality environment a firstdisplay region and a second display region separated from the firstdisplay region; determining a first duty cycle configuration for thefirst display region; determining a second duty cycle configuration forthe second display region, wherein the second duty cycle configurationdiffers from the first duty cycle configuration; and causing thewearable extended reality appliance to display the virtual content inaccordance with the determined first duty cycle configuration for thefirst display region and the determined second duty cycle configurationfor the second display region.

Some disclosed embodiments may include systems, methods, andnon-transitory computer readable media for extracting content from avirtual display. These embodiments may involve generating a virtualdisplay via a wearable extended reality appliance, wherein the virtualdisplay presents a group of virtual objects and is located at a firstvirtual distance from the wearable extended reality appliance;generating an extended reality environment via the wearable extendedreality appliance, wherein the extended reality environment includes atleast one additional virtual object presented at a second virtualdistance from the wearable extended reality appliance; receiving inputfor causing a specific virtual object from the group of virtual objectsto move from the virtual display to the extended reality environment;and in response to receiving the input, generating a presentation of aversion of the specific virtual object in the extended realityenvironment at a third virtual distance from the wearable extendedreality appliance, wherein the third virtual distance differs from thefirst virtual distance and the second virtual distance.

Some disclosed embodiments may include systems, methods, andnon-transitory computer readable media for selectively operating awearable extended reality appliance. These embodiments may involveestablishing a link between a wearable extended reality appliance and akeyboard device; receiving sensor data from at least one sensorassociated with the wearable extended reality appliance, the sensor databeing reflective of a relative orientation of the wearable extendedreality appliance with respect to the keyboard device; based on therelative orientation, selecting from a plurality of operation modes aspecific operation mode for the wearable extended reality appliance;identifying a user command based on at least one signal detected by thewearable extended reality appliance; and executing an action respondingto the identified user command in a manner consistent with the selectedoperation mode.

Some disclosed embodiments may include systems, methods, andnon-transitory computer readable media for generating videos ofindividuals interacting with virtual objects. These embodiments mayinvolve causing a wearable extended reality appliance to generate apresentation of an extended reality environment including at least onevirtual object; receiving first image data from at least a first imagesensor, the first image data reflecting a first perspective of anindividual wearing the wearable extended reality appliance; receivingsecond image data from at least a second image sensor, the second imagedata reflecting a second perspective facing the individual; identifyingin the first image data first physical hand movements interacting withthe at least one virtual object from the first perspective; identifyingin the second image data second physical hand movements interacting withthe at least one virtual object from the second perspective; analyzingat least one of the first image data or the second image data todetermine an interaction with the at least one virtual object; renderingfor display a representation of the at least one virtual object from thesecond perspective; and melding the rendered 1) representation of the atleast one virtual object from the second perspective with the secondimage data to generate a video of the individual interacting with the atleast one virtual object from the second perspective.

Some disclosed embodiments may include systems, methods, andnon-transitory computer readable media for enabling collaborationbetween physical writers and virtual writers. These embodiments mayinvolve receiving image data representing a hand of a first physicalwriter holding a physical marking implement and engaging with a physicalsurface to create tangible markings, wherein the image data is receivedfrom an image sensor associated with a wearable extended realityappliance worn by the first physical writer; transmitting informationbased on the image data to at least one computing device associated withat least one second virtual writer, to thereby enable the at least onesecond virtual writer to view the tangible markings created by the firstphysical writer; receiving from the at least one computing deviceannotation data representing additional markings in relative locationswith respect to the tangible markings created by the first physicalwriter; and in response to receiving the annotation data, causing thewearable extended reality appliance to overlay the physical surface withvirtual markings in the relative locations.

Some disclosed embodiments may include systems, methods, andnon-transitory computer readable media to tie at least one virtualspeaker to a physical space. These embodiments may involve receiving,via a wireless network, a first indication that a first wearableextended reality appliance is located in an area associated with avirtual speaker; transmitting to the first wearable extended realityappliance first data corresponding to first sounds associated with thevirtual speaker, to thereby enable a first user of the first wearableextended reality appliance to hear the first sounds during a first timeperiod, wherein the first sounds correspond to first settings of thevirtual speaker; receiving input associated with the first wearableextended reality appliance during the first time period, wherein thereceived input is indicative of second settings for the virtual speaker;transmitting to the first wearable extended reality appliance seconddata corresponding to second sounds associated with the virtual speaker,to thereby enable the first user of the first wearable extended realityappliance to hear the second sounds during a second time period, whereinthe second sounds correspond to the second settings of the virtualspeaker; after determining that the first user and the first wearableextended reality appliance left the area associated with the virtualspeaker, receiving, via the wireless network, a second indication that asecond wearable extended reality appliance is located in the areaassociated with the virtual speaker; and transmitting to the secondwearable extended reality appliance third data corresponding to thirdsounds associated with the virtual speaker, to thereby enable a seconduser of the second wearable extended reality appliance to hear the thirdsounds during a third time period, wherein the third sounds correspondto the second settings of the virtual speaker.

Some disclosed embodiments may include systems, methods, andnon-transitory computer readable media to initiate location-drivensensory prompts reflecting changes to virtual. These embodiments mayinvolve enabling interaction with a virtual object located in anextended reality environment associated with a wearable extended realityappliance; receiving data reflecting a change associated with thevirtual object; determining whether the virtual object is within a fieldof view of the wearable extended reality appliance or is outside thefield of view of the wearable extended reality appliance; causing thewearable extended reality appliance to initiate a first sensory promptindicative of the change associated with the virtual object when thevirtual object is determined to be within the field of view; and causingthe wearable extended reality appliance to initiate a second sensoryprompt indicative of the change associated with the virtual object whenthe virtual object is determined to be outside the field of view,wherein the second sensory prompt differs from the first sensory prompt.

Some disclosed embodiments may include systems, methods, andnon-transitory computer readable media for selectively controllingdisplay of digital objects. These embodiments may involve generating aplurality of digital objects for display in connection with use of acomputing device operable in a first display mode and in a seconddisplay mode, wherein in the first display mode, the plurality ofdigital objects are displayed via a physical display connected to thecomputing device, and in the second display mode, some of the pluralityof digital objects are displayed via the physical display, and at leastone other of the plurality of digital objects is displayed via awearable extended reality appliance; determining a usage status of thewearable extended reality appliance; selecting a display mode based onthe usage status of the wearable extended reality appliance; and inresponse to the display mode selection, outputting for presentation theplurality of digital objects in a manner consistent with the selecteddisplay mode.

Consistent with other disclosed embodiments, non-transitorycomputer-readable storage media may store program instructions, whichare executed by at least one processing device and perform any of themethods described herein.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various disclosed embodiments. Inthe drawings:

FIG. 1 is a schematic illustration of a user, using an example extendedreality system, consistent with some embodiments of the presentdisclosure.

FIG. 2 is a schematic illustration of the main components of the exampleextended reality system of FIG. 1 , consistent with some embodiments ofthe present disclosure.

FIG. 3 is a block diagram illustrating some of the components of aninput unit, consistent with some embodiments of the present disclosure.

FIG. 4 is a block diagram illustrating some of the components of anextended reality unit, consistent with some embodiments of the presentdisclosure.

FIG. 5 is a block diagram illustrating some of the components of aremote processing unit, consistent with some embodiments of the presentdisclosure.

FIG. 6 illustrates an exemplary extended reality environment fordisplaying virtual content, consistent with some embodiments of thepresent disclosure.

FIG. 7 illustrates another exemplary extended reality environment fordisplaying virtual content, consistent with some embodiments of thepresent disclosure.

FIG. 8 illustrates another exemplary extended reality environment fordisplaying virtual content, consistent with some embodiments of thepresent disclosure.

FIG. 9 is a block diagram illustrating an example process forcontrolling a duty cycle for a wearable extended reality appliance,consistent with embodiments of the present disclosure.

FIGS. 10 and 11 each illustrate an exemplary environment depicting awearer of a wearable extended reality appliance moving content between avirtual display and an extended reality environment, consistent withsome embodiments of the present disclosure.

FIG. 12 illustrates an exemplary environment depicting the wearer movinga window from a virtual display to an extended reality environment usinga control button, consistent with some embodiments of the presentdisclosure.

FIG. 13 illustrates an exemplary environment depicting the wearer movingcontent between the virtual display and the extended realityenvironment, consistent with some embodiments of the present disclosure.

FIG. 14 illustrates an exemplary environment depicting the wearer movingadditional content between the virtual display and the extended realityenvironment, consistent with some embodiments of the present disclosure.

FIG. 15 illustrates an exemplary environment depicting the wearerproviding a trigger to halt a presentation of content external to thevirtual display, consistent with some embodiments of the presentdisclosure.

FIG. 16 illustrates an exemplary environment depicting the wearer usinga keyboard to move content between the virtual display and the extendedreality environment, consistent with some embodiments of the presentdisclosure.

FIG. 17 illustrates an exemplary environment depicting the wearerviewing content as a thumbnail view, consistent with some embodiments ofthe present disclosure.

FIG. 18 is a block diagram illustrating an example process for movingcontent between a virtual display and an extended reality environment,consistent with some embodiments of the present disclosure.

FIG. 19 illustrates an exemplary environment depicting a wearer of thewearable extended reality appliance sitting adjacent to a keyboarddevice, consistent with some embodiments of the present disclosure.

FIG. 20 illustrates another exemplary environment depicting a wearer ofthe wearable extended reality appliance walking towards a keyboarddevice, consistent with some embodiments of the present disclosure.

FIG. 21 illustrates another exemplary environment depicting a wearer ofthe wearable extended reality appliance in proximity to and facing awayfrom a keyboard device, consistent with some embodiments of the presentdisclosure.

FIG. 22 illustrates another exemplary environment depicting a wearer ofthe wearable extended reality appliance walking towards a keyboarddevice and performing a gesture-based command, consistent with someembodiments of the present disclosure.

FIG. 23 illustrates another exemplary environment depicting a wearerinterfacing with the wearable extended reality appliance via audio whilefacing away from a keyboard device, consistent with some embodiments ofthe present disclosure.

FIG. 24 illustrates another exemplary environment depicting a wearer ofthe wearable extended reality appliance facing away from a keyboarddevice while a person approaches, consistent with some embodiments ofthe present disclosure.

FIG. 25 is a block diagram illustrating an example process forinterpreting commands in extended reality environments based ondistances from physical input devices, consistent with embodiments ofthe present disclosure.

FIG. 26 illustrates an exemplary wearable extended reality applianceincluding a first image sensor and a computing device including a secondimage sensor, consistent with some embodiments of the presentdisclosure.

FIG. 27 illustrates an exemplary view from the perspective of anindividual wearing the extended reality appliance, consistent with someembodiments of the present disclosure.

FIG. 28 illustrates an exemplary view from the perspective of a secondimage sensor facing an individual wearing the extended realityappliance, consistent with some embodiments of the present disclosure.

FIG. 29 illustrates exemplary virtual objects, consistent with someembodiments of the present disclosure.

FIG. 30 illustrates an exemplary melded view from the perspective of thesecond image sensor, consistent with some embodiments of the presentdisclosure.

FIG. 31 illustrates an exemplary melded view from the perspective of thesecond image sensor, consistent with disclosed embodiments.

FIG. 32 is a block diagram illustrating an example process forgenerating videos of individuals interacting with virtual objects,consistent with some embodiments of the present disclosure.

FIG. 33 is a schematic diagram illustrating use of an exemplary wearableextended reality appliance consistent with some disclosed embodiments.

FIG. 34 illustrates tangible markings on a physical surface, whichmarkings may be captured by a sensor of a wearable extended realitydevice consistent with some disclosed embodiments.

FIG. 35 illustrates the tangible markings of FIG. 34 , received by acomputing device consistent with some disclosed embodiments.

FIG. 36 illustrates the computing device of FIG. 35 , displayinglater-added additional markings, consistent with some disclosedembodiments.

FIG. 37 illustrates the additional markings displayed by the wearableextended reality device, consistent with some disclosed embodiments.

FIG. 38 is a block diagram illustrating an example process for virtualsharing of a physical surface consistent with some disclosedembodiments.

FIG. 39 illustrates an area in a physical space where a first user witha first wearable extended reality appliance is listening to audio atfirst settings of a virtual speaker, consistent with some embodiments ofthe present disclosure.

FIG. 40 illustrates the area in the physical space where the first userwith the first wearable extended reality appliance is listening to audioat second settings of the virtual speaker, consistent with someembodiments of the present disclosure.

FIG. 41 illustrates the area in the physical space where a second userwith a second wearable extended reality appliance is listening to audioat second settings of the virtual speaker, consistent with someembodiments of the present disclosure.

FIG. 42 illustrates the area in the physical space where the first userwith the first wearable extended reality appliance is listening to audioat the first settings of the virtual speaker and a location of thevirtual speaker in the area in the physical space has changed,consistent with some embodiments of the present disclosure.

FIG. 43 illustrates an area in a physical space with a directionalvirtual speaker positioned in a first orientation, consistent with someembodiments of the present disclosure.

FIG. 44 illustrates the area in the physical space with the directionalvirtual speaker positioned in a second orientation, consistent with someembodiments of the present disclosure.

FIG. 45 illustrates an area in a physical space with a virtual speakerhaving a sound zone of a first size and a user with a wearable extendedreality appliance is located in the sound zone, consistent with someembodiments of the present disclosure.

FIG. 46 illustrates an area in a physical space with a virtual speakerhaving a sound zone of a second size and the user with the wearableextended reality appliance is located outside the sound zone, consistentwith some embodiments of the present disclosure.

FIG. 47 illustrates an area in a physical space where a first user witha first wearable extended reality appliance and a second user with asecond wearable extended reality appliance are listening to audio atfirst settings of a virtual speaker, consistent with some embodiments ofthe present disclosure.

FIG. 48 is a block diagram illustrating an example process for tying avirtual speaker to a physical space, consistent with some embodiments ofthe present disclosure.

FIGS. 49, 50, 51, and 52 are schematic diagrams illustrating various usesnapshots of an example system for initiating sensory prompts forchanges based on a field of view consistent with some embodiments of thepresent disclosure.

FIG. 53 is a block diagram illustrating an example process forinitiating sensory prompts for changes based on a field of viewconsistent with some embodiments of the present disclosure.

FIG. 54 is a schematic illustration of a plurality of digital objectspresented to a user within an extended reality environment via aphysical display and a wearable extended reality appliance, consistentwith some embodiments of the present disclosure.

FIG. 55A is a schematic illustration of one example of a plurality ofdigital objects presented to a user in a first display mode, consistentwith some embodiments of the present disclosure.

FIG. 55B is a schematic illustration of one example of a plurality ofdigital objects presented to a user in a second display mode, consistentwith some embodiments of the present disclosure.

FIG. 56 is a schematic illustration of another example of a plurality ofdigital objects presented to a user in the second display mode,consistent with some embodiments of the present disclosure.

FIG. 57 is a block diagram illustrating an example process forselectively controlling a display of digital objects, consistent withsome embodiments of the present disclosure.

FIG. 58 is a block diagram illustrating an example process fordetermining a usage status of a wearable extended reality appliance,consistent with some embodiments of the present disclosure.

FIG. 59 is a block diagram illustrating an example process for selectinga display mode of a wearable extended reality appliance based on theusage status of the wearable extended reality appliance, consistent withsome embodiments of the present disclosure.

FIG. 60 is a block diagram illustrating an example process fordisplaying certain digital objects in the second display mode when thewearable extended reality appliance is in a second usage status,consistent with some embodiments of the present disclosure.

FIG. 61 is a block diagram illustrating an example process foridentifying a change in a usage status of a wearable extended realityappliance and revising the presentation of a plurality of digitalobjects, consistent with some embodiments of the present disclosure.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several illustrative embodiments are described herein,modifications, adaptations and other implementations are possible. Forexample, substitutions, additions, or modifications may be made to thecomponents illustrated in the drawings, and the illustrative methodsdescribed herein may be modified by substituting, reordering, removing,or adding steps to the disclosed methods. Accordingly, the followingdetailed description is not limited to the specific embodiments andexamples, but is inclusive of general principles described herein andillustrated in the figures in addition to the general principlesencompassed by the appended claims. The present disclosure is directedto systems and methods for providing users an extended realityenvironment. The term “extended reality environment,” which may also bereferred to as “extended reality,” “extended reality space.” or“extended environment,” refers to all types of real-and-virtual combinedenvironments and human-machine interactions at least partially generatedby computer technology. The extended reality environment may be acompletely simulated virtual environment or a combined real-and-virtualenvironment that a user may perceive from different perspectives. Insome examples, the user may interact with elements of the extendedreality environment. One non-limiting example of an extended realityenvironment may be a virtual reality environment, also known as “virtualreality” or a “virtual environment.” An immersive virtual realityenvironment may be a simulated non-physical environment which providesto the user the perception of being present in the virtual environment.Another non-limiting example of an extended reality environment may bean augmented reality environment, also known as “augmented reality” or“augmented environment.” An augmented reality environment may involvelive direct or indirect view of a physical real-world environment thatis enhanced with virtual computer-generated perceptual information, suchas virtual objects that the user may interact with. Another non-limitingexample of an extended reality environment is a mixed realityenvironment, also known as “mixed reality” or a “mixed environment.” Amixed reality environment may be a hybrid of physical real-world andvirtual environments, in which physical and virtual objects may coexistand interact in real time. In some examples, both augmented realityenvironments and mixed reality environments may include a combination ofreal and virtual worlds, real-time interactions, and accurate 3Dregistrations of virtual and real objects. In some examples, both theaugmented reality environment and the mixed reality environment mayinclude constructive overlaid sensory information that may be added tothe physical environment. In other examples, both the augmented realityenvironment and the mixed reality environment may include destructivevirtual content that may mask at least part of the physical environment.

In some embodiments, the systems and methods may provide the extendedreality environment using an extended reality appliance. The termextended reality appliance may include any type of device or system thatenables a user to perceive and/or interact with an extended realityenvironment. The extended reality appliance may enable the user toperceive and/or interact with an extended reality environment throughone or more sensory modalities. Some non-limiting examples of suchsensory modalities may include visual, auditory, haptic, somatosensory,and olfactory signals or feedback. One example of the extended realityappliance is a virtual reality appliance that enables the user toperceive and/or interact with a virtual reality environment. Anotherexample of the extended reality appliance is an augmented realityappliance that enables the user to perceive and/or interact with anaugmented reality environment. Yet another example of the extendedreality appliance is a mixed reality appliance that enables the user toperceive and/or interact with a mixed reality environment.

Consistent with one aspect of the disclosure, the extended realityappliance may be a wearable device, such as a head-mounted device, forexample, smart glasses, smart contact lens, headsets or any other deviceworn by a human for purposes of presenting an extended reality to thehuman. Other extended reality appliances may include a holographicprojector or any other device or system capable of providing anaugmented reality (AR), virtual reality (VR), mixed reality (MR), or anyimmersive experience. Typical components of wearable extended realityappliances may include at least one of: a stereoscopic head-mounteddisplay, a stereoscopic head-mounted sound system, head-motion trackingsensors (such as gyroscopes, accelerometers, magnetometers, imagesensors, structured light sensors, etc.), head mounted projectors,eye-tracking sensors, and/or additional components described below.Consistent with another aspect of the disclosure, the extended realityappliance may be a non-wearable extended reality appliance.Specifically, the non-wearable extended reality appliance may includemulti-projected environment appliances. In some embodiments, an extendedreality appliance may be configured to change the viewing perspective ofthe extended reality environment in response to movements of the userand in response to head movements of the user in particular. In oneexample, a wearable extended reality appliance may change thefield-of-view of the extended reality environment in response todetecting head movements and determining a change of the head pose ofthe user. The change the field-of-view of the extended realityenvironment may be achieved by changing the spatial orientation withoutchanging the spatial position of the user in the extended realityenvironment. In another example, a non-wearable extended realityappliance may change the spatial position of the user in the extendedreality environment in response to a change in the position of the userin the real world, for example, by changing the spatial position of theuser in the extended reality environment without changing the directionof the field-of-view with respect to the spatial position.

According to some embodiments, an extended reality appliance may includea digital communication device configured to at least one of: receivevirtual content data configured to enable a presentation of the virtualcontent, transmit virtual content for sharing with at least one externaldevice, receive contextual data from at least one external device,transmit contextual data to at least one external device, transmit usagedata indicative of usage of the extended reality appliance, and transmitdata based on information captured using at least one sensor included inthe extended reality appliance. In additional embodiments, the extendedreality appliance may include memory for storing at least one of virtualdata configured to enable a presentation of virtual content, contextualdata, usage data indicative of usage of the extended reality appliance,sensor data based on information captured using at least one sensorincluded in the extended reality appliance, software instructionsconfigured to cause a processing device to present the virtual content,software instructions configured to cause a processing device to collectand analyze the contextual data, software instructions configured tocause a processing device to collect and analyze the usage data, andsoftware instructions configured to cause a processing device to collectand analyze the sensor data. In additional embodiments, the extendedreality appliance may include a processing device configured to performat least one of rendering of virtual content, collecting and analyzingcontextual data, collecting and analyzing usage data, and collecting andanalyzing sensor data. In additional embodiments, the extended realityappliance may include one or more sensors. The one or more sensors mayinclude one or more image sensors (e.g., configured to capture imagesand/or videos of a user of the appliance or of an environment of theuser), one or more motion sensors (such as an accelerometer, agyroscope, a magnetometer, etc.), one or more positioning sensors (suchas GPS, outdoor positioning sensor, indoor positioning sensor, etc.),one or more temperature sensors (e.g., configured to measure thetemperature of at least part of the appliance and/or of theenvironment), one or more contact sensors, one or more proximity sensors(e.g., configured to detect whether the appliance is currently worn),one or more electrical impedance sensors (e.g., configured to measureelectrical impedance of the user), one or more eye tracking sensors,such as gaze detectors, optical trackers, electric potential trackers(e.g., electrooculogram (EOG) sensors), video-based eye-trackers,infra-red/near infra-red sensors, passive light sensors, or any othertechnology capable of determining where a human is looking or gazing.

In some embodiments, the systems and methods may use an input device tointeract with the extended reality appliance. The term input device mayinclude any physical device configured to receive input from a user oran environment of the user, and to provide the data to a computationaldevice. The data provided to the computational device may be in adigital format and/or in an analog format. In one embodiment, the inputdevice may store the input received from the user in a memory deviceaccessible by a processing device, and the processing device may accessthe stored data for analysis. In another embodiment, the input devicemay provide the data directly to a processing device, for example, overa bus or over another communication system configured to transfer datafrom the input device to the processing device. In some examples, theinput received by the input device may include key presses, tactileinput data, motion data, position data, gestures based input data,direction data, or any other data.

Some examples of the input device may include a button, a key, akeyboard, a computer mouse, a touchpad, a touchscreen, a joystick, oranother mechanism from which input may be received. Another example ofan input device may include an integrated computational interface devicethat includes at least one physical component for receiving input from auser. The integrated computational interface device may include at leasta memory, a processing device, and the at least one physical componentfor receiving input from a user. In one example, the integratedcomputational interface device may further include a digital networkinterface that enables digital communication with other computingdevices. In one example, the integrated computational interface devicemay further include a physical component for outputting information tothe user. In some examples, all components of the integratedcomputational interface device may be included in a single housing,while in other examples the components may be distributed among two ormore housings. Some non-limiting examples of physical components forreceiving input from users that may be included in the integratedcomputational interface device may include at least one of a button, akey, a keyboard, a touchpad, a touchscreen, a joystick, or any othermechanism or sensor from which computational information may bereceived. Some non-limiting examples of physical components foroutputting information to users may include at least one of a lightindicator (such as a LED indicator), a screen, a touchscreen, a beeper,an audio speaker, or any other audio, video, or haptic device thatprovides human-perceptible outputs.

In some embodiments, image data may be captured using one or more imagesensors. In some examples, the image sensors may be included in theextended reality appliance, in a wearable device, in the wearableextended reality device, in the input device, in an environment of auser, and so forth. In some examples, the image data may be read frommemory, may be received from an external device, may be generated (forexample, using a generative model), and so forth. Some non-limitingexamples of image data may include images, grayscale images, colorimages, 2D images, 3D images, videos, 2D videos, 3D videos, frames,footages, data derived from other image data, and so forth. In someexamples, the image data may be encoded in any analog or digital format.Some non-limiting examples of such formats may include raw formats,compressed formats, uncompressed formats, lossy formats, losslessformats, JPEG, GIF, PNG, TIFF, BMP, NTSC, PAL, SECAM, MPEG, MPEG-4 Part14, MOV, WMV, FLV, AVI, AVCHD, WebM, MKV, and so forth.

In some embodiments, the extended reality appliance may receive digitalsignals, for example, from the input device. The term digital signalsrefers to a series of digital values that are discrete in time. Thedigital signals may represent, for example, sensor data, textual data,voice data, video data, virtual data, or any other form of data thatprovides perceptible information. Consistent with the presentdisclosure, the digital signals may be configured to cause the extendedreality appliance to present virtual content. In one embodiment, thevirtual content may be presented in a selected orientation. In thisembodiment, the digital signals may indicate a position and an angle ofa viewpoint in an environment, such as an extended reality environment.Specifically, the digital signals may include an encoding of theposition and angle in six degree-of-freedom coordinates (e.g.,forward/back, up/down, left/right, yaw, pitch, and roll). In anotherembodiment, the digital signals may include an encoding of the positionas three-dimensional coordinates (e.g., x, y, and z), and an encoding ofthe angle as a vector originating from the encoded position.Specifically, the digital signals may indicate the orientation and anangle of the presented virtual content in absolute coordinates of theenvironment, for example, by encoding yaw, pitch and roll of the virtualcontent with respect to a standard default angle. In another embodiment,the digital signals may indicate the orientation and the angle of thepresented virtual content with respect to a viewpoint of another object(e.g., a virtual object, a physical object, etc.), for example, byencoding yaw, pitch, and roll of the virtual content with respect to adirection corresponding to the viewpoint or to a direction correspondingto the other object. In another embodiment, such digital signals mayinclude one or more projections of the virtual content, for example, ina format ready for presentation (e.g., image, video, etc.). For example,each such projection may correspond to a particular orientation or aparticular angle. In another embodiment, the digital signals may includea representation of virtual content, for example, by encoding objects ina three-dimensional array of voxels, in a polygon mesh, or in any otherformat in which virtual content may be presented.

In some embodiments, the digital signals may be configured to cause theextended reality appliance to present virtual content. The term virtualcontent may include any type of data representation that may bedisplayed by the extended reality appliance to the user. The virtualcontent may include a virtual object, inanimate virtual content, animatevirtual content configured to change over time or in response totriggers, virtual two-dimensional content, virtual three dimensionalcontent, a virtual overlay over a portion of a physical environment orover a physical object, a virtual addition to a physical environment orto a physical object, a virtual promotion content, a virtualrepresentation of a physical object, a virtual representation of aphysical environment, a virtual document, a virtual character orpersona, a virtual computer screen, a virtual widget, or any otherformat for displaying information virtually. Consistent with the presentdisclosure, the virtual content may include any visual presentationrendered by a computer or a processing device. In one embodiment, thevirtual content may include a virtual object that is a visualpresentation rendered by a computer in a confined region and configuredto represent an object of a particular type (such as an inanimatevirtual object, an animate virtual object, virtual furniture, a virtualdecorative object, virtual widget, or other virtual representation.).The rendered visual presentation may change to reflect changes to astatus of an object or changes in the viewing angle of the object, forexample, in a way that mimics changes in the appearance of physicalobjects. In another embodiment, the virtual content may include avirtual display (also referred to as a “virtual display screen” or a“virtual screen” herein), such as a virtual computer screen, a virtualtablet screen or a virtual smartphone screen, configured to displayinformation generated by an operating system, in which the operatingsystem may be configured to receive textual data from a physicalkeyboard and/or a virtual keyboard and to cause a display of the textualcontent in the virtual display screen. In one example, illustrated inFIG. 1 , the virtual content may include a virtual environment thatincludes a virtual computer screen and a plurality of virtual objects.In some examples, a virtual display may be a virtual object mimickingand/or extending the functionality of a physical display screen. Forexample, the virtual display may be presented in an extended realityenvironment (such as a mixed reality environment, an augmented realityenvironment, a virtual reality environment, etc.), using an extendedreality appliance. In one example, a virtual display may present contentproduced by a regular operating system that may be equally presented ona physical display screen. In one example, a textual content enteredusing a keyboard (for example, using a physical keyboard, using avirtual keyboard, etc.) may be presented on a virtual display in realtime as the textual content is typed. In one example, a virtual cursormay be presented on a virtual display, and the virtual cursor may becontrolled by a pointing device (such as a physical pointing device, avirtual pointing device, a computer mouse, a joystick, a touchpad, aphysical touch controller, and/or any other device for identifying alocation on the display). In one example, one or more windows of agraphical user interface operating system may be presented on a virtualdisplay. In another example, content presented on a virtual display maybe interactive, that is, it may change in reaction to actions of users.In yet another example, a presentation of a virtual display may includea presentation of a screen frame, or may include no presentation of ascreen frame.

Some disclosed embodiments may include and/or access a data structure ora database. The terms data structure and a database, consistent with thepresent disclosure may include any collection of data values andrelationships among them. The data may be stored linearly, horizontally,hierarchically, relationally, non-relationally, uni-dimensionally,multidimensionally, operationally, in an ordered manner, in an unorderedmanner, in an object-oriented manner, in a centralized manner, in adecentralized manner, in a distributed manner, in a custom manner, or inany manner enabling data access. By way of non-limiting examples, datastructures may include an array, an associative array, a linked list, abinary tree, a balanced tree, a heap, a stack, a queue, a set, a hashtable, a record, a tagged union, Entity-Relationship model, a graph, ahypergraph, a matrix, a tensor, and/or other ways of organizing data.For example, a data structure may include an XML database, an RDBMSdatabase, an SQL database or NoSQL alternatives for data storage/searchsuch as, for example, MongoDB, Redis, Couchbase, Datastax EnterpriseGraph, Elastic Search, Splunk, Solr, Cassandra, Amazon DynamoDB, Scylla,HBase, and/or Neo4J. A data structure may be a component of thedisclosed system or a remote computing component (e.g., a cloud-baseddata structure). Data in the data structure may be stored in contiguousor non-contiguous memory. Moreover, a data structure may not requireinformation to be co-located. It may be distributed across multipleservers, for example, the multiple servers may be owned or operated bythe same or different entities. Thus, the term data structure in thesingular is inclusive of plural data structures.

In some embodiments, the system may determine the confidence level inreceived input or in any determined value. The term confidence levelrefers to any indication, numeric or otherwise, of a level (e.g., withina predetermined range) indicative of an amount of confidence the systemhas in the determined data. For example, the confidence level may have avalue between 1 and 10. Alternatively, the confidence level may beexpressed as a percentage or any other numerical or non-numericalindication. In some cases, the system may compare the confidence levelto a threshold. The term threshold may denote a reference value, alevel, a point, or a range of values. In operation, when the confidencelevel of determined data exceeds the threshold (or is below it,depending on a particular use case), the system may follow a firstcourse of action and, when the confidence level is below it (or aboveit, depending on a particular use case), the system may follow a secondcourse of action. The value of the threshold may be predetermined foreach type of examined object or may be dynamically selected based ondifferent considerations.

System Overview

Reference is now made to FIG. 1 , which illustrates a user that uses anexample extended reality system consistent with various embodiments ofthe present disclosure. FIG. 1 is an exemplary representation of justone embodiment, and it is to be understood that some illustratedelements might be omitted and others added within the scope of thisdisclosure. As shown, a user 100 is sitting behind table 102, supportinga keyboard 104 and mouse 106. Keyboard 104 is connected by wire 108 to awearable extended reality appliance 110 that displays virtual content touser 100. Alternatively or additionally, keyboard 104 may connect towearable extended reality appliance 110 wirelessly. For illustrationpurposes, the wearable extended reality appliance is depicted as a pairof smart glasses, but, as described above, wearable extended realityappliance 110 may be any type of head-mounted device used for presentingan extended reality to user 100. The virtual content displayed bywearable extended reality appliance 110 includes a virtual screen 112(also referred to as a “virtual display screen” or a “virtual display”herein) and a plurality of virtual widgets 114. Virtual widgets114A-114D are displayed next to virtual screen 112 and virtual widget114E is displayed on table 102. User 100 may input text to a document116 displayed in virtual screen 112 using keyboard 104, and may controlvirtual cursor 118 using mouse 106. In one example, virtual cursor 118may move anywhere within virtual screen 112. In another example, virtualcursor 118 may move anywhere within virtual screen 112 and may also moveto any one of virtual widgets 114A-114D but not to virtual widget 114E.In yet another example, virtual cursor 118 may move anywhere withinvirtual screen 112 and may also move to any one of virtual widgets114A-114E. In an additional example, virtual cursor 118 may moveanywhere in the extended reality environment including virtual screen112 and virtual widgets 114A-114E. In yet another example, virtualcursor may move on all available surfaces (i.e., virtual surfaces orphysical surfaces) or only on selected surfaces in the extended realityenvironment. Alternatively or additionally, user 100 may interact withany one of virtual widgets 114A-114E, or with selected virtual widgets,using hand gestures recognized by wearable extended reality appliance110. For example, virtual widget 114E may be an interactive widget(e.g., a virtual slider controller) that may be operated with handgestures.

FIG. 2 illustrates an example of a system 200 that provides extendedreality (XR) experience to users, such as user 100. FIG. 2 is anexemplary representation of just one embodiment, and it is to beunderstood that some illustrated elements might be omitted and othersadded within the scope of this disclosure. System 200 may becomputer-based and may include computer system components, wearableappliances, workstations, tablets, handheld computing devices, memorydevices, and/or internal network(s) connecting the components. System200 may include or be connected to various network computing resources(e.g., servers, routers, switches, network connections, storage devices)for supporting services provided by system 200. Consistent with thepresent disclosure, system 200 may include an input unit 202, an XR unit204, a mobile communications device 206, and/or a remote processing unit208. Remote processing unit 208 may include a server 210 coupled to oneor more physical or virtual storage devices, such as a data structure212. System 200 may also include or be connected to a communicationsnetwork 214 that facilitates communications and data exchange betweendifferent system components and the different entities associated withsystem 200.

Consistent with the present disclosure, input unit 202 may include oneor more devices that may receive input from user 100. In one embodiment,input unit 202 may include a textual input device, such as keyboard 104.The textual input device may include all possible types of devices andmechanisms for inputting textual information to system 200. Examples oftextual input devices may include mechanical keyboards, membranekeyboards, flexible keyboards, QWERTY keyboards, Dvorak keyboards,Colemak keyboards, chorded keyboards, wireless keyboards, keypads,key-based control panels, or other arrays of control keys, vision inputdevices, or any other mechanism for inputting text, whether themechanism is provided in physical form or is presented virtually. In oneembodiment, input unit 202 may also include a pointing input device,such as mouse 106. The pointing input device may include all possibletypes of devices and mechanisms for inputting two-dimensional orthree-dimensional information to system 200. In one example,two-dimensional input from the pointing input device may be used forinteracting with virtual content presented via the XR unit 204. Examplesof pointing input devices may include a computer mouse, trackball,touchpad, trackpad, touchscreen, joystick, pointing stick, stylus, lightpen, or any other physical or virtual input mechanism. In oneembodiment, input unit 202 may also include a graphical input device,such as a touchscreen configured to detect contact, movement, or breakof movement. The graphical input device may use any of a plurality oftouch sensitivity technologies, including, but not limited to,capacitive, resistive, infrared, and surface acoustic wave technologiesas well as other proximity sensor arrays or other elements fordetermining one or more points of contact. In one embodiment, input unit202 may also include one or more voice input devices, such as amicrophone. The voice input device may include all possible types ofdevices and mechanisms for inputting voice data to facilitatevoice-enabled functions, such as voice recognition, voice replication,digital recording, and telephony functions. In one embodiment, inputunit 202 may also include one or more image input devices, such as animage sensor, configured to capture image data. In one embodiment, inputunit 202 may also include one or more haptic gloves configured tocapture hands motion and pose data. In one embodiment, input unit 202may also include one or more proximity sensors configured to detectpresence and/or movement of objects in a selected region near thesensors.

In accordance with some embodiments, the system may include at least onesensor configured to detect and/or measure a property associated withthe user, the user's action, or user's environment. One example of theat least one sensor, is sensor 216 included in input unit 202. Sensor216 may be a motion sensor, a touch sensor, a light sensor, an infraredsensor, an audio sensor, an image sensor, a proximity sensor, apositioning sensor, a gyroscope, a temperature sensor, a biometricsensor, or any other sensing devices to facilitate relatedfunctionalities. Sensor 216 may be integrated with, or connected to, theinput devices or it may be separated from the input devices. In oneexample, a thermometer may be included in mouse 106 to determine thebody temperature of user 100. In another example, a positioning sensormay be integrated with keyboard 104 to determine movement of user 100relative to keyboard 104. Such positioning sensor may be implementedusing one of the following technologies: Global Positioning System(GPS), GLObal NAvigation Satellite System (GLONASS), Galileo globalnavigation system, BeiDou navigation system, other Global NavigationSatellite Systems (GNSS), Indian Regional Navigation Satellite System(IRNSS), Local Positioning Systems (LPS), Real-Time Location Systems(RTLS), Indoor Positioning System (IPS), Wi-Fi based positioningsystems, cellular triangulation, image based positioning technology,indoor positioning technology, outdoor positioning technology, or anyother positioning technology.

In accordance with some embodiments, the system may include one or moresensors for identifying a position and/or a movement of a physicaldevice (such as a physical input device, a physical computing device,keyboard 104, mouse 106, wearable extended reality appliance 110, and soforth). The one or more sensors may be included in the physical deviceor may be external to the physical device. In some examples, an imagesensor external to the physical device (for example, an image sensorincluded in another physical device) may be used to capture image dataof the physical device, and the image data may be analyzed to identifythe position and/or the movement of the physical device. For example,the image data may be analyzed using a visual object tracking algorithmto identify the movement of the physical device, may be analyzed using avisual object detection algorithm to identify the position of thephysical device (for example, relative to the image sensor, in a globalcoordinates system, etc.), and so forth. In some examples, an imagesensor included in the physical device may be used to capture imagedata, and the image data may be analyzed to identify the position and/orthe movement of the physical device. For example, the image data may beanalyzed using visual odometry algorithms to identify the position ofthe physical device, may be analyzed using an ego-motion algorithm toidentify movement of the physical device, and so forth. In someexamples, a positioning sensor, such as an indoor positioning sensor oran outdoor positioning sensor, may be included in the physical deviceand may be used to determine the position of the physical device. Insome examples, a motion sensor, such as an accelerometer or a gyroscope,may be included in the physical device and may be used to determine themotion of the physical device. In some examples, a physical device, suchas a keyboard or a mouse, may be configured to be positioned on aphysical surface. Such physical device may include an optical mousesensor (also known as non-mechanical tracking engine) aimed towards thephysical surface, and the output of the optical mouse sensor may beanalyzed to determine movement of the physical device with respect tothe physical surface.

Consistent with the present disclosure, XR unit 204 may include awearable extended reality appliance configured to present virtualcontent to user 100. One example of the wearable extended realityappliance is wearable extended reality appliance 110. Additionalexamples of wearable extended reality appliance may include a VirtualReality (VR) device, an Augmented Reality (AR) device, a Mixed Reality(MR) device, or any other device capable of generating extended realitycontent. Some non-limiting examples of such devices may include NrealLight, Magic Leap One, Varjo, Quest 1/2, Vive, and others. In someembodiments, XR unit 204 may present virtual content to user 100.Generally, an extended reality appliance may include allreal-and-virtual combined environments and human-machine interactionsgenerated by computer technology and wearables. As mentioned above, theterm “extended reality” (XR) refers to a superset which includes theentire spectrum from “the complete real” to “the complete virtual.” Itincludes representative forms such as augmented reality (AR), mixedreality (MR), virtual reality (VR), and the areas interpolated amongthem. Accordingly, it is noted that the terms “XR appliance,” “ARappliance,” “VR appliance,” and “MR appliance” may be usedinterchangeably herein and may refer to any device of the variety ofappliances listed above.

Consistent with the present disclosure, the system may exchange datawith a variety of communication devices associated with users, forexample, mobile communications device 206. The term “communicationdevice” is intended to include all possible types of devices capable ofexchanging data using digital communications network, analogcommunication network or any other communications network configured toconvey data. In some examples, the communication device may include asmartphone, a tablet, a smartwatch, a personal digital assistant, adesktop computer, a laptop computer, an IoT device, a dedicatedterminal, a wearable communication device, and any other device thatenables data communications. In some cases, mobile communications device206 may supplement or replace input unit 202. Specifically, mobilecommunications device 206 may be associated with a physical touchcontroller that may function as a pointing input device. Moreover,mobile communications device 206 may also, for example, be used toimplement a virtual keyboard and replace the textual input device. Forexample, when user 100 steps away from table 102 and walks to the breakroom with his smart glasses, he may receive an email that requires aquick answer. In this case, the user may select to use his or her ownsmartwatch as the input device and to type the answer to the email whileit is virtually presented by the smart glasses.

Consistent with the present disclosure, embodiments of the system mayinvolve the usage of a cloud server. The term “cloud server” refers to acomputer platform that provides services via a network, such as theInternet. In the example embodiment illustrated in FIG. 2 , server 210may use virtual machines that may not correspond to individual hardware.For example, computational and/or storage capabilities may beimplemented by allocating appropriate portions of desirablecomputation/storage power from a scalable repository, such as a datacenter or a distributed computing environment. Specifically, in oneembodiment, remote processing unit 208 may be used together with XR unit204 to provide the virtual content to user 100. In one exampleconfiguration, server 210 may be a cloud server that functions as theoperating system (OS) of the wearable extended reality appliance. In oneexample, server 210 may implement the methods described herein usingcustomized hard-wired logic, one or more Application Specific IntegratedCircuits (ASICs), Field Programmable Gate Arrays (FPGAs), firmware,and/or program logic which, in combination with the computer system,cause server 210 to be a special-purpose machine.

In some embodiments, server 210 may access data structure 212 todetermine, for example, virtual content to display to user 100. Datastructure 212 may utilize a volatile or non-volatile, magnetic,semiconductor, tape, optical, removable, non-removable, other type ofstorage device or tangible or non-transitory computer-readable medium,or any medium or mechanism for storing information. Data structure 212may be part of server 210 or separate from server 210, as shown. Whendata structure 212 is not part of server 210, server 210 may exchangedata with data structure 212 via a communication link. Data structure212 may include one or more memory devices that store data andinstructions used to perform one or more features of the disclosedmethods. In one embodiment, data structure 212 may include any of aplurality of suitable data structures, ranging from small datastructures hosted on a workstation to large data structures distributedamong data centers. Data structure 212 may also include any combinationof one or more data structures controlled by memory controller devices(e.g., servers) or software.

Consistent with the present disclosure, communications network may beany type of network (including infrastructure) that supportscommunications, exchanges information, and/or facilitates the exchangeof information between the components of a system. For example,communications network 214 in system 200 may include, for example, atelephone network, an extranet, an intranet, the Internet, satellitecommunications, off-line communications, wireless communications,transponder communications, a Local Area Network (LAN), wireless network(e.g., a Wi-Fi/302.11 network), a Wide Area Network (WAN), a VirtualPrivate Network (VPN), digital communication network, analogcommunication network, or any other mechanism or combination ofmechanisms that enables data transmission. The components andarrangements of system 200 shown in FIG. 2 are intended to be exemplaryonly and are not intended to limit any embodiment, as the systemcomponents used to implement the disclosed processes and features mayvary.

FIG. 3 is a block diagram of an exemplary configuration of input unit202. FIG. 3 is an exemplary representation of just one embodiment, andit is to be understood that some illustrated elements might be omittedand others added within the scope of this disclosure. In the embodimentof FIG. 3 , input unit 202 may directly or indirectly access a bus 300(or other communication mechanism) that interconnects subsystems andcomponents for transferring information within input unit 202. Forexample, bus 300 may interconnect a memory interface 310, a networkinterface 320, an input interface 330, a power source 340, an outputinterface 350, a processing device 360, a sensors interface 370, and adatabase 380.

Memory interface 310, shown in FIG. 3 , may be used to access a softwareproduct and/or data stored on a non-transitory computer-readable medium.Generally, a non-transitory computer-readable storage medium refers toany type of physical memory on which information or data readable by atleast one processor can be stored. Examples include Random Access Memory(RAM), Read-Only Memory (ROM), volatile memory, nonvolatile memory, harddrives, CD ROMs, DVDs, flash drives, disks, any other optical datastorage medium, any physical medium with patterns of holes, a PROM, anEPROM, a FLASH-EPROM or any other flash memory, NVRAM, a cache, aregister, any other memory chip or cartridge, and networked versions ofthe same. The terms “memory” and “computer-readable storage medium” mayrefer to multiple structures, such as a plurality of memories orcomputer-readable storage mediums located within an input unit or at aremote location. Additionally, one or more computer-readable storagemediums can be utilized in implementing a computer-implemented method.Accordingly, the term computer-readable storage medium should beunderstood to include tangible items and exclude carrier waves andtransient signals. In the specific embodiment illustrated in FIG. 3 ,memory interface 310 may be used to access a software product and/ordata stored on a memory device, such as memory device 311. Memory device311 may include high-speed random-access memory and/or non-volatilememory, such as one or more magnetic disk storage devices, one or moreoptical storage devices, and/or flash memory (e.g., NAND, NOR).Consistent with the present disclosure, the components of memory device311 may be distributed in more than units of system 200 and/or in morethan one memory device.

Memory device 311, shown in FIG. 3 , may contain software modules toexecute processes consistent with the present disclosure. In particular,memory device 311 may include an input determination module 312, anoutput determination module 313, a sensors communication module 314, avirtual content determination module 315, a virtual contentcommunication module 316, and a database access module 317. Modules312-317 may contain software instructions for execution by at least oneprocessor (e.g., processing device 360) associated with input unit 202.Input determination module 312, output determination module 313, sensorscommunication module 314, virtual content determination module 315,virtual content communication module 316, and database access module 317may cooperate to perform various operations. For example, inputdetermination module 312 may determine text using data received from,for example, keyboard 104. Thereafter, output determination module 313may cause presentation of the recent inputted text, for example on adedicated display 352 physically or wirelessly coupled to keyboard 104.This way, when user 100 types, the user can see a preview of the typedtext without constantly moving his head up and down to look at virtualscreen 112. Sensors communication module 314 may receive data fromdifferent sensors to determine a status of user 100. Thereafter, virtualcontent determination module 315 may determine the virtual content todisplay, based on received input and the determined status of user 100.For example, the determined virtual content may be a virtualpresentation of the recent inputted text on a virtual screen virtuallylocated adjacent to keyboard 104. Virtual content communication module316 may obtain virtual content that is not determined by virtual contentdetermination module 315 (e.g., an avatar of another user). Theretrieval of the virtual content may be from database 380, from remoteprocessing unit 208, or any other source.

In some embodiments, input determination module 312 may regulate theoperation of input interface 330 in order to receive pointer input 331,textual input 332, audio input 333, and XR-related input 334. Details onthe pointer input, the textual input, and the audio input are describedabove. The term “XR-related input” may include any type of data that maycause a change in the virtual content displayed to user 100. In oneembodiment, XR-related input 334 may include image data of user 100 fromthe wearable extended reality appliance (e.g., detected hand gestures ofuser 100). In another embodiment, XR-related input 334 may includewireless communication indicating a presence of another user inproximity to user 100. Consistent with the present disclosure, inputdetermination module 312 may concurrently receive different types ofinput data. Thereafter, input determination module 312 may further applydifferent rules based on the detected type of input. For example, apointer input may have precedence over voice input.

In some embodiments, output determination module 313 may regulate theoperation of output interface 350 in order to generate output usinglight indicators 351, display 352, and/or speakers 353. In general, theoutput generated by output determination module 313 does not includevirtual content to be presented by a wearable extended realityappliance. Instead, the output generated by output determination module313 includes various outputs that relates to the operation of input unit202 and/or the operation of XR unit 204. In one embodiment, lightindicators 351 may include a light indicator that shows the status of awearable extended reality appliance. For example, the light indicatormay display green light when wearable extended reality appliance 110 areconnected to keyboard 104, and blinks when wearable extended realityappliance 110 has low battery. In another embodiment, display 352 may beused to display operational information. For example, the display maypresent error messages when the wearable extended reality appliance isinoperable. In another embodiment, speakers 353 may be used to outputaudio, for example, when user 100 wishes to play some music for otherusers.

In some embodiments, sensors communication module 314 may regulate theoperation of sensors interface 370 in order to receive sensor data fromone or more sensors, integrated with, or connected to, an input device.The one or more sensors may include: audio sensor 371, image sensor 372,motion sensor 373, environmental sensor 374 (e.g., a temperature sensor,ambient light detectors, etc.), and other sensors 375. In oneembodiment, the data received from sensors communication module 314 maybe used to determine the physical orientation of the input device. Thephysical orientation of the input device may be indicative of a state ofthe user and may be determined based on combination of a tilt movement,a roll movement, and a lateral movement. Thereafter, the physicalorientation of the input device may be used by virtual contentdetermination module 315 to modify display parameters of the virtualcontent to match the state of the user (e.g., attention, sleepy, active,sitting, standing, leaning backwards, leaning forward, walking, moving,riding).

In some embodiments, virtual content determination module 315 maydetermine the virtual content to be displayed by the wearable extendedreality appliance. The virtual content may be determined based on datafrom input determination module 312, sensors communication module 314,and other sources (e.g., database 380). In some embodiments, determiningthe virtual content may include determining the distance, the size, andthe orientation of the virtual objects. The determination of theposition of the virtual objects may be determined based on the type ofthe virtual objects. Specifically, with regards to the exampleillustrated in FIG. 1 , the virtual content determination module 315 maydetermine to place four virtual widgets 114A-114D on the sides ofvirtual screen 112 and to place virtual widget 114E on table 102 becausevirtual widget 114E is a virtual controller (e.g., volume bar). Thedetermination of the position of the virtual objects may further bedetermined based on user's preferences. For example, for left-handedusers, virtual content determination module 315 may determine placing avirtual volume bar left of keyboard 104; and for right-handed users,virtual content determination module 315 may determine placing thevirtual volume bar right of keyboard 104.

In some embodiments, virtual content communication module 316 mayregulate the operation of network interface 320 in order to obtain datafrom one or more sources to be presented as virtual content to user 100.The one or more sources may include other XR units 204, the user'smobile communications device 206, remote processing unit 208, publiclyavailable information, etc. In one embodiment, virtual contentcommunication module 316 may communicate with mobile communicationsdevice 206 in order to provide a virtual representation of mobilecommunications device 206. For example, the virtual representation mayenable user 100 to read messages and interact with applicationsinstalled on the mobile communications device 206. Virtual contentcommunication module 316 may also regulate the operation of networkinterface 320 in order to share virtual content with other users. In oneexample, virtual content communication module 316 may use data frominput determination module to identify a trigger (e.g., the trigger mayinclude a gesture of the user) and to transfer content from the virtualdisplay to a physical display (e.g., TV) or to a virtual display of adifferent user.

In some embodiments, database access module 317 may cooperate withdatabase 380 to retrieve stored data. The retrieved data may include,for example, privacy levels associated with different virtual objects,the relationship between virtual objects and physical objects, theuser's preferences, the user's past behavior, and more. As describedabove, virtual content determination module 315 may use the data storedin database 380 to determine the virtual content. Database 380 mayinclude separate databases, including, for example, a vector database,raster database, tile database, viewport database, and/or a user inputdatabase. The data stored in database 380 may be received from modules314-317 or other components of system 200. Moreover, the data stored indatabase 380 may be provided as input using data entry, data transfer,or data uploading.

Modules 312-317 may be implemented in software, hardware, firmware, amix of any of those, or the like. In some embodiments, any one or moreof modules 312-317 and data associated with database 380 may be storedin XR unit 204, mobile communications device 206, or remote processingunit 208. Processing devices of system 200 may be configured to executethe instructions of modules 312-317. In some embodiments, aspects ofmodules 312-317 may be implemented in hardware, in software (includingin one or more signal processing and/or application specific integratedcircuits), in firmware, or in any combination thereof, executable by oneor more processors, alone, or in various combinations with each other.Specifically, modules 312-317 may be configured to interact with eachother and/or other modules of system 200 to perform functions consistentwith some disclosed embodiments. For example, input unit 202 may executeinstructions that include an image processing algorithm on data from XRunit 204 to determine head movement of user 100. Furthermore, eachfunctionality described throughout the specification, with regards toinput unit 202 or with regards to a component of input unit 202, maycorrespond to a set of instructions for performing said functionality.These instructions need not be implemented as separate softwareprograms, procedures, or modules. Memory device 311 may includeadditional modules and instructions or fewer modules and instructions.For example, memory device 311 may store an operating system, such asANDROID, iOS, UNIX, OSX, WINDOWS, DARWIN, RTXC, LINUX or an embeddedoperating system such as VXWorkS. The operating system can includeinstructions for handling basic system services and for performinghardware-dependent tasks.

Network interface 320, shown in FIG. 3 , may provide two-way datacommunications to a network, such as communications network 214. In oneembodiment, network interface 320 may include an Integrated ServicesDigital Network (ISDN) card, cellular modem, satellite modem, or a modemto provide a data communication connection over the Internet. As anotherexample, network interface 320 may include a Wireless Local Area Network(WLAN) card. In another embodiment, network interface 320 may include anEthernet port connected to radio frequency receivers and transmittersand/or optical (e.g., infrared) receivers and transmitters. The specificdesign and implementation of network interface 320 may depend on thecommunications network or networks over which input unit 202 is intendedto operate. For example, in some embodiments, input unit 202 may includenetwork interface 320 designed to operate over a GSM network, a GPRSnetwork, an EDGE network, a Wi-Fi or WiMax network, and a Bluetoothnetwork. In any such implementation, network interface 320 may beconfigured to send and receive electrical, electromagnetic, or opticalsignals that carry digital data streams or digital signals representingvarious types of information.

Input interface 330, shown in FIG. 3 , may receive input from a varietyof input devices, for example, a keyboard, a mouse, a touch pad, a touchscreen, one or more buttons, a joystick, a microphone, an image sensor,and any other device configured to detect physical or virtual input. Thereceived input may be in the form of at least one of: text, sounds,speech, hand gestures, body gestures, tactile information, and any othertype of physically or virtually input generated by the user. In thedepicted embodiment, input interface 330 may receive pointer input 331,textual input 332, audio input 333, and XR-related input 334. Inadditional embodiments, input interface 330 may be an integrated circuitthat may act as a bridge between processing device 360 and any of theinput devices listed above.

Power source 340, shown in FIG. 3 , may provide electrical energy topower input unit 202 and optionally also power XR unit 204. Generally, apower source included in the any device or system in the presentdisclosure may be any device that can repeatedly store, dispense, orconvey electric power, including, but not limited to, one or morebatteries (e.g., a lead-acid battery, a lithium-ion battery, anickel-metal hydride battery, a nickel-cadmium battery), one or morecapacitors, one or more connections to external power sources, one ormore power convertors, or any combination of them. With reference to theexample illustrated in FIG. 3 , the power source may be mobile, whichmeans that input unit 202 can be easily carried by hand (e.g., the totalweight of power source 340 may be less than a pound). The mobility ofthe power source enables user 100 to use input unit 202 in a variety ofsituations. In other embodiments, power source 340 may be associatedwith a connection to an external power source (such as an electricalpower grid) that may be used to charge power source 340. In addition,power source 340 may be configured to charge one or more batteriesincluded in XR unit 204; for example, a pair of extended reality glasses(e.g., wearable extended reality appliance 110) may be charged (e.g.,wirelessly or not wirelessly) when they are placed on or in proximity tothe input unit 202.

Output interface 350, shown in FIG. 3 , may cause output from a varietyof output devices, for example, using light indicators 351, display 352,and/or speakers 353. In one embodiment, output interface 350 may be anintegrated circuit that may act as bridge between processing device 360and at least one of the output devices listed above. Light indicators351 may include one or more light sources, for example, a LED arrayassociated with different colors. Display 352 may include a screen(e.g., LCD or dot-matrix screen) or a touch screen. Speakers 353 mayinclude audio headphones, a hearing aid type device, a speaker, a boneconduction headphone, interfaces that provide tactile cues, and/orvibrotactile stimulators.

Processing device 360, shown in FIG. 3 , may include at least oneprocessor configured to execute computer programs, applications,methods, processes, or other software to perform embodiments describedin the present disclosure. Generally, a processing device included inany device or system in the present disclosure may include one or moreintegrated circuits, microchips, microcontrollers, microprocessors, allor part of a central processing unit (CPU), graphics processing unit(GPU), digital signal processor (DSP), field programmable gate array(FPGA), or other circuits suitable for executing instructions orperforming logic operations. The processing device may include at leastone processor configured to perform functions of the disclosed methodssuch as a microprocessor manufactured by Intel™. The processing devicemay include a single core or multiple core processors executing parallelprocesses simultaneously. In one example, the processing device may be asingle core processor configured with virtual processing technologies.The processing device may implement virtual machine technologies orother technologies to provide the ability to execute, control, run,manipulate, store, etc., multiple software processes, applications,programs, etc. In another example, the processing device may include amultiple-core processor arrangement (e.g., dual, quad core, etc.)configured to provide parallel processing functionalities to allow adevice associated with the processing device to execute multipleprocesses simultaneously. Other types of processor arrangements may beimplemented to provide the capabilities disclosed herein.

Sensors interface 370, shown in FIG. 3 , may obtain sensor data from avariety of sensors, for example, audio sensor 371, image sensor 372,motion sensor 373, environmental sensor 374, and other sensors 375. Inone embodiment, sensors interface 370 may be an integrated circuit thatmay act as bridge between processing device 360 and at least one of thesensors listed above.

Audio sensor 371 may include one or more audio sensors configured tocapture audio by converting sounds to digital information. Some examplesof audio sensors may include: microphones, unidirectional microphones,bidirectional microphones, cardioid microphones, omnidirectionalmicrophones, onboard microphones, wired microphones, wirelessmicrophones, or any combination of the above. Consistent with thepresent disclosure, processing device 360 may modify a presentation ofvirtual content based on data received from audio sensor 371 (e.g.,voice commands).

Image sensor 372 may include one or more image sensors configured tocapture visual information by converting light to image data. Consistentwith the present disclosure, an image sensor may be included in the anydevice or system in the present disclosure and may be any device capableof detecting and converting optical signals in the near-infrared,infrared, visible, and ultraviolet spectrums into electrical signals.Examples of image sensors may include digital cameras, phone cameras,semiconductor Charge-Coupled Devices (CCDs), active pixel sensors inComplementary Metal-Oxide-Semiconductor (CMOS), or N-typemetal-oxide-semiconductor (NMOS, Live MOS). The electrical signals maybe used to generate image data. Consistent with the present disclosure,the image data may include pixel data streams, digital images, digitalvideo streams, data derived from captured images, and data that may beused to construct one or more 3D images, a sequence of 3D images, 3Dvideos, or a virtual 3D representation. The image data acquired by imagesensor 372 may be transmitted by wired or wireless transmission to anyprocessing device of system 200. For example, the image data may beprocessed in order to: detect objects, detect events, detect actions,detect faces, detect people, recognize a known person, or determine anyother information that may be used by system 200. Consistent with thepresent disclosure, processing device 360 may modify a presentation ofvirtual content based on image data received from image sensor 372.

Motion sensor 373 may include one or more motion sensors configured tomeasure motion of input unit 202 or motion of objects in the environmentof input unit 202. Specifically, the motion sensors may perform at leastone of the following: detect motion of objects in the environment ofinput unit 202, measure the velocity of objects in the environment ofinput unit 202, measure the acceleration of objects in the environmentof input unit 202, detect the motion of input unit 202, measure thevelocity of input unit 202, and/or measure the acceleration of inputunit 202. In some embodiments, motion sensor 373 may include one or moreaccelerometers configured to detect changes in proper accelerationand/or to measure proper acceleration of input unit 202. In otherembodiments, motion sensor 373 may include one or more gyroscopesconfigured to detect changes in the orientation of input unit 202 and/orto measure information related to the orientation of input unit 202. Inother embodiments, motion sensor 373 may include one or more imagesensors, LIDAR sensors, radar sensors, or proximity sensors. Forexample, by analyzing captured images the processing device maydetermine the motion of input unit 202, for example, using ego-motionalgorithms. In addition, the processing device may determine the motionof objects in the environment of input unit 202, for example, usingobject tracking algorithms. Consistent with the present disclosure,processing device 360 may modify a presentation of virtual content basedon the determined motion of input unit 202 or the determined motion ofobjects in the environment of input unit 202. For example, causing avirtual display to follow the movement of input unit 202.

Environmental sensor 374 may include one or more sensors from differenttypes configured to capture data reflective of the environment of inputunit 202. In some embodiments, environmental sensor 374 may include oneor more chemical sensors configured to perform at least one of thefollowing: measure chemical properties in the environment of input unit202, measure changes in the chemical properties in the environment ofinput unit 202, detect the present of chemicals in the environment ofinput unit 202, measure the concentration of chemicals in theenvironment of input unit 202. Examples of such chemical properties mayinclude: pH level, toxicity, and temperature. Examples of such chemicalsmay include: electrolytes, particular enzymes, particular hormones,particular proteins, smoke, carbon dioxide, carbon monoxide, oxygen,ozone, hydrogen, and hydrogen sulfide. In other embodiments,environmental sensor 374 may include one or more temperature sensorsconfigured to detect changes in the temperature of the environment ofinput unit 202 and/or to measure the temperature of the environment ofinput unit 202. In other embodiments, environmental sensor 374 mayinclude one or more barometers configured to detect changes in theatmospheric pressure in the environment of input unit 202 and/or tomeasure the atmospheric pressure in the environment of input unit 202.In other embodiments, environmental sensor 374 may include one or morelight sensors configured to detect changes in the ambient light in theenvironment of input unit 202. Consistent with the present disclosure,processing device 360 may modify a presentation of virtual content basedon input from environmental sensor 374. For example, automaticallyreducing the brightness of the virtual content when the environment ofuser 100 becomes darker.

Other sensors 375 may include a weight sensor, a light sensor, aresistive sensor, an ultrasonic sensor, a proximity sensor, a biometricsensor, or other sensing devices to facilitate related functionalities.In some embodiments, other sensors 375 may include one or morepositioning sensors configured to obtain positioning information ofinput unit 202, to detect changes in the position of input unit 202,and/or to measure the position of input unit 202. Alternatively, GPSsoftware may permit input unit 202 to access an external GPS receiver(e.g., connecting via a serial port or Bluetooth). Consistent with thepresent disclosure, processing device 360 may modify a presentation ofvirtual content based on input from other sensors 375. For example,presenting private information only after identifying user 100 usingdata from a biometric sensor.

The components and arrangements shown in FIG. 3 are not intended tolimit any embodiment. As will be appreciated by a person skilled in theart having the benefit of this disclosure, numerous variations and/ormodifications may be made to the depicted configuration of input unit202. For example, not all components may be essential for the operationof an input unit in all cases. Any component may be located in anyappropriate part of an input unit, and the components may be rearrangedinto a variety of configurations while providing the functionality ofvarious embodiments. For example, some input units may not include allof the elements as shown in input unit 202.

FIG. 4 is a block diagram of an exemplary configuration of XR unit 204.FIG. 4 is an exemplary representation of just one embodiment, and it isto be understood that some illustrated elements might be omitted andothers added within the scope of this disclosure. In the embodiment ofFIG. 4 , XR unit 204 may directly or indirectly access a bus 400 (orother communication mechanism) that interconnects subsystems andcomponents for transferring information within XR unit 204. For example,bus 400 may interconnect a memory interface 410, a network interface420, an input interface 430, a power source 440, an output interface450, a processing device 460, a sensors interface 470, and a database480.

Memory interface 410, shown in FIG. 4 , is assumed to have similarfunctionality as the functionality of memory interface 310 describedabove in detail. Memory interface⁴¹⁰ may be used to access a softwareproduct and/or data stored on a non-transitory computer-readable mediumor on memory devices, such as memory device 411. Memory device 411 maycontain software modules to execute processes consistent with thepresent disclosure. In particular, memory device 411 may include aninput determination module 412, an output determination module 413, asensors communication module 414, a virtual content determination module415, a virtual content communication module 416, and a database accessmodule 417. Modules 412-417 may contain software instructions forexecution by at least one processor (e.g., processing device 460)associated with XR unit 204. Input determination module 412, outputdetermination module 413, sensors communication module 414, virtualcontent determination module 415, virtual content communication module416, and database access module 417 may cooperate to perform variousoperations. For example, input determination module 412 may determineUser Interface (UI) input received from input unit 202. At the sametime, sensors communication module 414 may receive data from differentsensors to determine a status of user 100. Virtual content determinationmodule 415 may determine the virtual content to display based onreceived input and the determined status of user 100. Virtual contentcommunication module 416 may retrieve virtual content not determined byvirtual content determination module 415. The retrieval of the virtualcontent may be from database 380, database 480, mobile communicationsdevice 206, or from remote processing unit 208. Based on the output ofvirtual content determination module 415, output determination module413 may cause a change in a virtual content displayed to user 100 byprojector 454.

In some embodiments, input determination module 412 may regulate theoperation of input interface 430 in order to receive gesture input 431,virtual input 432, audio input 433, and UI input 434. Consistent withthe present disclosure, input determination module 412 may concurrentlyreceive different types of input data. In one embodiment, inputdetermination module 412 may apply different rules based on the detectedtype of input. For example, gesture input may have precedence overvirtual input. In some embodiments, output determination module 413 mayregulate the operation of output interface 450 in order to generateoutput using light indicators 451, display 452, speakers 453, andprojector 454. In one embodiment, light indicators 451 may include alight indicator that shows the status of the wearable extended realityappliance. For example, the light indicator may display green light whenthe wearable extended reality appliance 110 are connected to input unit202, and blinks when wearable extended reality appliance 110 has lowbattery. In another embodiment, display 452 may be used to displayoperational information. In another embodiment, speakers 453 may includea bone conduction headphone used to output audio to user 100. In anotherembodiment, projector 454 may present virtual content to user 100.

The operations of a sensors communication module, a virtual contentdetermination module, a virtual content communication module, and adatabase access module are described above with reference to FIG. 3 ,details of which are not repeated herein. Modules 412-417 may beimplemented in software, hardware, firmware, a mix of any of those, orthe like.

Network interface 420, shown in FIG. 4 , is assumed to have similarfunctionality as the functionality of network interface 320, describedabove in detail. The specific design and implementation of networkinterface 420 may depend on the communications network(s) over which XRunit 204 is intended to operate. For example, in some embodiments. XRunit 204 is configured to be selectively connectable by wire to inputunit 202. When connected by wire, network interface 420 may enablecommunications with input unit 202; and when not connected by wire,network interface 420 may enable communications with mobilecommunications device 206.

Input interface 430, shown in FIG. 4 , is assumed to have similarfunctionality as the functionality of input interface 330 describedabove in detail. In this case, input interface 430 may communicate withan image sensor to obtain gesture input 431 (e.g., a finger of user 100pointing to a virtual object), communicate with other XR units 204 toobtain virtual input 432 (e.g., a virtual object shared with XR unit 204or a gesture of avatar detected in the virtual environment), communicatewith a microphone to obtain audio input 433 (e.g., voice commands), andcommunicate with input unit 202 to obtain UI input 434 (e.g., virtualcontent determined by virtual content determination module 315).

Power source 440, shown in FIG. 4 , is assumed to have similarfunctionality as the functionality of power source 340 described above,only it provides electrical energy to power XR unit 204. In someembodiments, power source 440 may be charged by power source 340. Forexample, power source 440 may be wirelessly changed when XR unit 204 isplaced on or in proximity to input unit 202.

Output interface 450, shown in FIG. 4 , is assumed to have similarfunctionality as the functionality of output interface 350 describedabove in detail. In this case, output interface 450 may cause outputfrom light indicators 451, display 452, speakers 453, and projector 454.Projector 454 may be any device, apparatus, instrument, or the likecapable of projecting (or directing) light in order to display virtualcontent onto a surface. The surface may be part of XR unit 204, part ofan eye of user 100, or part of an object in proximity to user 100. Inone embodiment, projector 454 may include a lighting unit thatconcentrates light within a limited solid angle by means of one or moremirrors and lenses, and may provide a high value of luminous intensityin a defined direction.

Processing device 460, shown in FIG. 4 , is assumed to have similarfunctionality as the functionality of processing device 360 describedabove in detail. When XR unit 204 is connected to input unit 202,processing device 460 may work together with processing device 360.Specifically, processing device 460 may implement virtual machinetechnologies or other technologies to provide the ability to execute,control, run, manipulate, store, etc., multiple software processes,applications, programs, etc. It is appreciated that other types ofprocessor arrangements could be implemented to provide the capabilitiesdisclosed herein.

Sensors interface 470, shown in FIG. 4 , is assumed to have similarfunctionality as the functionality of sensors interface 370 describedabove in detail. Specifically, sensors interface 470 may communicatewith audio sensor 471, image sensor 472, motion sensor 473,environmental sensor 474, and other sensors 475. The operations of anaudio sensor, an image sensor, a motion sensor, an environmental sensor,and other sensors are described above with reference to FIG. 3 , detailsof which are not repeated herein. It will be appreciated that othertypes and combination of sensors may be used to provide the capabilitiesdisclosed herein.

The components and arrangements shown in FIG. 4 are not intended tolimit any embodiment. As will be appreciated by a person skilled in theart having the benefit of this disclosure, numerous variations and/ormodifications may be made to the depicted configuration of XR unit 204.For example, not all components may be essential for the operation of XRunit 204 in all cases. Any component may be located in any appropriatepart of system 200, and the components may be rearranged into a varietyof configurations while providing the functionality of variousembodiments. For example, some XR units may not include all of theelements in XR unit 204 (e.g., wearable extended reality appliance 110may not have light indicators 451).

FIG. 5 is a block diagram of an exemplary configuration of remoteprocessing unit 208. FIG. 5 is an exemplary representation of just oneembodiment, and it is to be understood that some illustrated elementsmight be omitted and others added within the scope of this disclosure.In the embodiment of FIG. 5 , remote processing unit 208 may include aserver 210 that directly or indirectly accesses a bus 500 (or othercommunication mechanism) interconnecting subsystems and components fortransferring information within server 210. For example, bus 500 mayinterconnect a memory interface 510, a network interface 520, a powersource 540, a processing device 560, and a database 580. Remoteprocessing unit 208 may also include a one or more data structures. Forexample, data structures 212A, 212B, and 212C.

Memory interface 510, shown in FIG. 5 , is assumed to have similarfunctionality as the functionality of memory interface 310 describedabove in detail. Memory interface 510 may be used to access a softwareproduct and/or data stored on a non-transitory computer-readable mediumor on other memory devices, such as memory devices 311, 411, 511, ordata structures 212A, 212B, and 212C. Memory device 511 may containsoftware modules to execute processes consistent with the presentdisclosure. In particular, memory device 511 may include a shared memorymodule 512, a node registration module 513, a load balancing module 514,one or more computational nodes 515, an internal communication module516, an external communication module 517, and a database access module(not shown). Modules 512-517 may contain software instructions forexecution by at least one processor (e.g., processing device 560)associated with remote processing unit 208. Shared memory module 512,node registration module 513, load balancing module 514, computationalmodule 515, and external communication module 517 may cooperate toperform various operations.

Shared memory module 512 may allow information sharing between remoteprocessing unit 208 and other components of system 200. In someembodiments, shared memory module 512 may be configured to enableprocessing device 560 (and other processing devices in system 200) toaccess, retrieve, and store data. For example, using shared memorymodule 512, processing device 560 may perform at least one of: executingsoftware programs stored on memory device 511, database 580, or datastructures 212A-C; storing information in memory device 511, database580, or data structures 212A-C; or retrieving information from memorydevice 511, database 580, or data structures 212A-C.

Node registration module 513 may be configured to track the availabilityof one or more computational nodes 515. In some examples, noderegistration module 513 may be implemented as: a software program, suchas a software program executed by one or more computational nodes 515, ahardware solution, or a combined software and hardware solution. In someimplementations, node registration module 513 may communicate with oneor more computational nodes 515, for example, using internalcommunication module 516. In some examples, one or more computationalnodes 515 may notify node registration module 513 of their status, forexample, by sending messages: at startup, at shutdown, at constantintervals, at selected times, in response to queries received from noderegistration module 513, or at any other determined times. In someexamples, node registration module 513 may query about the status of oneor more computational nodes 515, for example, by sending messages: atstartup, at constant intervals, at selected times, or at any otherdetermined times.

Load balancing module 514 may be configured to divide the workload amongone or more computational nodes 515. In some examples, load balancingmodule 514 may be implemented as: a software program, such as a softwareprogram executed by one or more of the computational nodes 515, ahardware solution, or a combined software and hardware solution. In someimplementations, load balancing module 514 may interact with noderegistration module 513 in order to obtain information regarding theavailability of one or more computational nodes 515. In someimplementations, load balancing module 514 may communicate with one ormore computational nodes 515, for example, using internal communicationmodule 516. In some examples, one or more computational nodes 515 maynotify load balancing module 514 of their status, for example, bysending messages: at startup, at shutdown, at constant intervals, atselected times, in response to queries received from load balancingmodule 514, or at any other determined times. In some examples, loadbalancing module 514 may query about the status of one or morecomputational nodes 515, for example, by sending messages: at startup,at constant intervals, at pre-selected times, or at any other determinedtimes.

Internal communication module 516 may be configured to receive and/or totransmit information from one or more components of remote processingunit 208. For example, control signals and/or synchronization signalsmay be sent and/or received through internal communication module 516.In one embodiment, input information for computer programs, outputinformation of computer programs, and/or intermediate information ofcomputer programs may be sent and/or received through internalcommunication module 516. In another embodiment, information receivedthough internal communication module 516 may be stored in memory device511, in database 580, in data structures 212A-C, or other memory devicein system 200. For example, information retrieved from data structure212A may be transmitted using internal communication module 516. Inanother example, input data may be received using internal communicationmodule 516 and stored in data structure 212B.

External communication module 517 may be configured to receive and/or totransmit information from one or more components of system 200. Forexample, control signals may be sent and/or received through externalcommunication module 517. In one embodiment, information received thoughexternal communication module 517 may be stored in memory device 511, indatabase 580, in data structures 212A-C, and or any memory device in thesystem 200. In another embodiment, information retrieved from any ofdata structures 212A-C may be transmitted using external communicationmodule 517 to XR unit 204. In another embodiment, input data may betransmitted and/or received using external communication module 517.Examples of such input data may include data received from input unit202, information captured from the environment of user 100 using one ormore sensors (e.g., audio sensor 471, image sensor 472, motion sensor473, environmental sensor 474, other sensors 475), and more.

In some embodiments, aspects of modules 512-517 may be implemented inhardware, in software (including in one or more signal processing and/orapplication specific integrated circuits), in firmware, or in anycombination thereof, executable by one or more processors, alone, or invarious combinations with each other. Specifically, modules 512-517 maybe configured to interact with each other and/or other modules of system200 to perform functions consistent with disclosed embodiments. Memorydevice 511 may include additional modules and instructions or fewermodules and instructions.

Network interface 520, power source 540, processing device 560, anddatabase 580, shown in FIG. 5 , are assumed to have similarfunctionality as the functionality of similar elements described abovewith reference to FIGS. 4 and 5 . The specific design and implementationof the above-mentioned components may vary based on the implementationof system 200. In addition, remote processing unit 208 may include moreor fewer components. For example, remote processing unit 208 may includean input interface configured to receive direct input from one or moreinput devices.

Consistent with the present disclosure, a processing device of system200 (e.g., processor within mobile communications device 206, aprocessor within a server 210, a processor within a wearable extendedreality appliance, such as, wearable extended reality appliance 110,and/or a processor within an input device associated with wearableextended reality appliance 110, such as keyboard 104) may use machinelearning algorithms in order to implement any of the methods disclosedherein. In some embodiments, machine learning algorithms (also referredto as machine learning models in the present disclosure) may be trainedusing training examples, for example in the cases described below. Somenon-limiting examples of such machine learning algorithms may includeclassification algorithms, data regressions algorithms, imagesegmentation algorithms, visual detection algorithms (such as objectdetectors, face detectors, person detectors, motion detectors, edgedetectors, etc.), visual recognition algorithms (such as facerecognition, person recognition, object recognition, etc.), speechrecognition algorithms, mathematical embedding algorithms, naturallanguage processing algorithms, support vector machines, random forests,nearest neighbors algorithms, deep learning algorithms, artificialneural network algorithms, convolutional neural network algorithms,recurrent neural network algorithms, linear machine learning models,non-linear machine learning models, ensemble algorithms, and more. Forexample, a trained machine learning algorithm may comprise an inferencemodel, such as a predictive model, a classification model, a dataregression model, a clustering model, a segmentation model, anartificial neural network (such as a deep neural network, aconvolutional neural network, a recurrent neural network, etc.), arandom forest, a support vector machine, and so forth. In some examples,the training examples may include example inputs together with thedesired outputs corresponding to the example inputs. Further, in someexamples, training machine learning algorithms using the trainingexamples may generate a trained machine learning algorithm, and thetrained machine learning algorithm may be used to estimate outputs forinputs not included in the training examples. In some examples,engineers, scientists, processes and machines that train machinelearning algorithms may further use validation examples and/or testexamples. For example, validation examples and/or test examples mayinclude example inputs together with the desired outputs correspondingto the example inputs, a trained machine learning algorithm and/or anintermediately trained machine learning algorithm may be used toestimate outputs for the example inputs of the validation examplesand/or test examples, the estimated outputs may be compared to thecorresponding desired outputs, and the trained machine learningalgorithm and/or the intermediately trained machine learning algorithmmay be evaluated based on a result of the comparison. In some examples,a machine learning algorithm may have parameters and hyper parameters,where the hyper parameters may be set manually by a person orautomatically by a process external to the machine learning algorithm(such as a hyper parameter search algorithm), and the parameters of themachine learning algorithm may be set by the machine learning algorithmbased on the training examples. In some implementations, thehyper-parameters may be set based on the training examples and thevalidation examples, and the parameters may be set based on the trainingexamples and the selected hyper-parameters. For example, given thehyper-parameters, the parameters may be conditionally independent of thevalidation examples.

In some embodiments, trained machine learning algorithms (also referredto as machine learning models and trained machine learning models in thepresent disclosure) may be used to analyze inputs and generate outputs,for example in the cases described below. In some examples, a trainedmachine learning algorithm may be used as an inference model that whenprovided with an input generates an inferred output. For example, atrained machine learning algorithm may include a classificationalgorithm, the input may include a sample, and the inferred output mayinclude a classification of the sample (such as an inferred label, aninferred tag, and so forth). In another example, a trained machinelearning algorithm may include a regression model, the input may includea sample, and the inferred output may include an inferred valuecorresponding to the sample. In yet another example, a trained machinelearning algorithm may include a clustering model, the input may includea sample, and the inferred output may include an assignment of thesample to at least one cluster. In an additional example, a trainedmachine learning algorithm may include a classification algorithm, theinput may include an image, and the inferred output may include aclassification of an item depicted in the image. In yet another example,a trained machine learning algorithm may include a regression model, theinput may include an image, and the inferred output may include aninferred value corresponding to an item depicted in the image (such asan estimated property of the item, such as size, volume, age of a persondepicted in the image, distance from an item depicted in the image, andso forth). In an additional example, a trained machine learningalgorithm may include an image segmentation model, the input may includean image, and the inferred output may include a segmentation of theimage. In yet another example, a trained machine learning algorithm mayinclude an object detector, the input may include an image, and theinferred output may include one or more detected objects in the imageand/or one or more locations of objects within the image. In someexamples, the trained machine learning algorithm may include one or moreformulas and/or one or more functions and/or one or more rules and/orone or more procedures, the input may be used as input to the formulasand/or functions and/or rules and/or procedures, and the inferred outputmay be based on the outputs of the formulas and/or functions and/orrules and/or procedures (for example, selecting one of the outputs ofthe formulas and/or functions and/or rules and/or procedures, using astatistical measure of the outputs of the formulas and/or functionsand/or rules and/or procedures, and so forth).

Consistent with the present disclosure, a processing device of system200 may analyze image data captured by an image sensor (e.g., imagesensor 372, image sensor 472, or any other image sensor) in order toimplement any of the methods disclosed herein. In some embodiments,analyzing the image data may comprise analyzing the image data to obtaina preprocessed image data, and subsequently analyzing the image dataand/or the preprocessed image data to obtain the desired outcome. One ofordinary skill in the art will recognize that the followings areexamples, and that the image data may be preprocessed using other kindsof preprocessing methods. In some examples, the image data may bepreprocessed by transforming the image data using a transformationfunction to obtain a transformed image data, and the preprocessed imagedata may comprise the transformed image data. For example, thetransformed image data may comprise one or more convolutions of theimage data. For example, the transformation function may comprise one ormore image filters, such as low-pass filters, high-pass filters,band-pass filters, all-pass filters, and so forth. In some examples, thetransformation function may comprise a nonlinear function. In someexamples, the image data may be preprocessed by smoothing at least partsof the image data, for example using Gaussian convolution, using amedian filter, and so forth. In some examples, the image data may bepreprocessed to obtain a different representation of the image data. Forexample, the preprocessed image data may comprise: a representation ofat least part of the image data in a frequency domain; a DiscreteFourier Transform of at least part of the image data; a Discrete WaveletTransform of at least part of the image data; a time/frequencyrepresentation of at least part of the image data; a representation ofat least part of the image data in a lower dimension; a lossyrepresentation of at least part of the image data; a losslessrepresentation of at least part of the image data; a time ordered seriesof any of the above; any combination of the above; and so forth. In someexamples, the image data may be preprocessed to extract edges, and thepreprocessed image data may comprise information based on and/or relatedto the extracted edges. In some examples, the image data may bepreprocessed to extract image features from the image data. Somenon-limiting examples of such image features may comprise informationbased on and/or related to: edges; corners; blobs; ridges; ScaleInvariant Feature Transform (SIFT) features; temporal features; and soforth. In some examples, analyzing the image data may includecalculating at least one convolution of at least a portion of the imagedata, and using the calculated at least one convolution to calculate atleast one resulting value and/or to make determinations,identifications, recognitions, classifications, and so forth.

Consistent with other aspects of the disclosure, a processing device ofsystem 200 may analyze image data in order to implement any of themethods disclosed herein. In some embodiments, analyzing the image maycomprise analyzing the image data and/or the preprocessed image datausing one or more rules, functions, procedures, artificial neuralnetworks, object detection algorithms, face detection algorithms, visualevent detection algorithms, action detection algorithms, motiondetection algorithms, background subtraction algorithms, inferencemodels, and so forth. Some non-limiting examples of such inferencemodels may include: an inference model preprogrammed manually; aclassification model; a regression model; a result of trainingalgorithms, such as machine learning algorithms and/or deep learningalgorithms, on training examples, where the training examples mayinclude examples of data instances, and in some cases, a data instancemay be labeled with a corresponding desired label and/or result, andmore. In some embodiments, analyzing image data (for example by themethods, steps and modules described herein) may comprise analyzingpixels, voxels, point cloud, range data, etc. included in the imagedata.

A convolution may include a convolution of any dimension. Aone-dimensional convolution is a function that transforms an originalsequence of numbers to a transformed sequence of numbers. Theone-dimensional convolution may be defined by a sequence of scalars.Each particular value in the transformed sequence of numbers may bedetermined by calculating a linear combination of values in asubsequence of the original sequence of numbers corresponding to theparticular value. A result value of a calculated convolution may includeany value in the transformed sequence of numbers. Likewise, ann-dimensional convolution is a function that transforms an originaln-dimensional array to a transformed array. The n-dimensionalconvolution may be defined by an n-dimensional array of scalars (knownas the kernel of the n-dimensional convolution). Each particular valuein the transformed array may be determined by calculating a linearcombination of values in an n-dimensional region of the original arraycorresponding to the particular value. A result value of a calculatedconvolution may include any value in the transformed array. In someexamples, an image may comprise one or more components (such as colorcomponents, depth component, etc.), and each component may include a twodimensional array of pixel values. In one example, calculating aconvolution of an image may include calculating a two dimensionalconvolution on one or more components of the image. In another example,calculating a convolution of an image may include stacking arrays fromdifferent components to create a three dimensional array, andcalculating a three dimensional convolution on the resulting threedimensional array. In some examples, a video may comprise one or morecomponents (such as color components, depth component, etc.), and eachcomponent may include a three dimensional array of pixel values (withtwo spatial axes and one temporal axis). In one example, calculating aconvolution of a video may include calculating a three dimensionalconvolution on one or more components of the video. In another example,calculating a convolution of a video may include stacking arrays fromdifferent components to create a four dimensional array, and calculatinga four dimensional convolution on the resulting four dimensional array.

Wearable extended reality appliances may include different displayregions. The display regions may be permanently set, or may dynamicallyconfigurable, for example by software or hardware components.Dynamically controlling the display luminance or intensity in thedifferent display regions may be beneficial, for example to conserveresources, and/or accommodate user visibility needs. For example, thedisplay luminance may be dimmed in a less relevant region of a displayand intensified in a more relevant region to hold the focus of the useron the more relevant region, and/or to efficiently allocate resources(e.g., electrical energy). As another example, the display luminance maybe dimmed or intensified to prevent eye strain, motion sickness, toaccommodate ambient lighting conditions, and/or energy consumptionrequirements. One technique for dynamically controlling the displayluminance may be to dynamically control the duty cycle of the displaysignal in each region of a wearable extended reality appliance. In otherexamples, dynamically controlling the duty cycle of the display signalin each region of a wearable extended reality appliance may bebeneficial regardless of the display luminance or intensity, for exampleto prevent eye strain, motion sickness, to accommodate ambient lightingconditions, and/or energy consumption requirements. In some examples,dynamically controlling the duty cycle of the display signal in theentire display of a wearable extended reality appliance may bebeneficial, for example to prevent eye strain, motion sickness, toaccommodate ambient lighting conditions, and/or energy consumptionrequirements.

In some embodiments, duty cycle control operations may be performed forwearable extended reality appliances. Data representing virtual contentin an extended reality environment and associated with a wearableextended reality appliance may be received. Two separate display regions(e.g., a first display region and a second display region), of thewearable extended reality appliance may be identified. A duty cycleconfiguration may be determined for each display region. Thus, a firstduty cycle configuration may be determined for the first display region,and a second duty cycle configuration may be determined for the seconddisplay region, where the second duty cycle configuration differs fromthe first duty cycle configuration. The wearable extended realityappliance may be caused to display virtual content in the first displayregion according to the first determined duty cycle configuration and inthe second display region according to the second determined duty cycleconfiguration. In this manner, virtual content may be displayed in eachdisplay region of the wearable extended reality appliance in accordancewith a different duty cycle configuration.

In some instances, the description that follows may refer to FIGS. 6-9which illustrate exemplary implementations for performing duty cyclecontrol operations for representing virtual content in an extendedreality environment associated with a wearable extended realityappliance, consistent with some disclosed embodiments. FIGS. 6-9 areintended merely to facilitate the conceptualizing of one exemplaryimplementation for performing duty cycle control operations to representvirtual content via a wearable extended reality appliance and do notlimit the disclosure to any particular implementation. Additionally,while the description that follows generally relates to a first dutycycle configuration corresponding to a higher duty cycle and a secondduty cycle configuration corresponding to a lower duty cycle, this isfor illustrative purposes only and does not limit the invention. It thusmay be noted that some implementations and/or applications may includethe first duty cycle configuration corresponding to a lower duty cyclethan the second duty cycle configuration. Additionally, the use of thedescriptors “first’ and “second” is intended merely to distinguishbetween two different entities and does not necessarily assign a higherordinality or importance to one entity versus the other. The descriptionthat follows includes references to smart glasses as an exemplaryimplementation of a wearable extended reality appliance. It is to beunderstood that these examples are merely intended to assist in gaininga conceptual understanding of disclosed embodiments, and do not limitthe disclosure to any particular implementation for a wearable extendedreality appliance. The disclosure is thus understood to relate to anyimplementation for a wearable extended reality appliance, includingimplementations different than smart glasses.

Some embodiments involve a non-transitory computer readable mediumcontaining instructions that when executed by at least one processorcause the at least one processor to perform duty cycle controloperations for wearable extended reality appliances. The term“non-transitory computer-readable medium” may be understood as describedearlier. The term “instructions” may refer to program code instructionsthat may be executed by a computer processor. The instructions may bewritten in any type of computer programming language, such as aninterpretive language (e.g., scripting languages such as HTML andJavaScript), a procedural or functional language (e.g., C or Pascal thatmay be compiled for converting to executable code), object-orientedprogramming language (e.g., Java or Python), logical programminglanguage (e.g., Prolog or Answer Set Programming), or any otherprogramming language. In some embodiments, the instructions mayimplement methods associated with machine learning, deep learning,artificial intelligence, digital image processing, optimizationalgorithms, and any other computer processing technique. The term“processor” may refer to any physical device having an electric circuitthat performs a logic operation. A processor may include one or moreintegrated circuits, microchips, microcontrollers, microprocessors, allor part of a central processing unit (CPU), graphics processing unit(GPU), digital signal processor (DSP), field programmable gate array(FPGA), or other circuits suitable for executing instructions orperforming logic operations, as described earlier.

The term “cycle” may refer to a portion of an oscillating signal thatrepeats periodically (e.g., regularly) over time. For each cycle of anoscillating signal, a fraction of the cycle may be associated with a“high” (e.g., on or active) state, another fraction of the cycle may beassociated with a “low” (e.g., off or inactive) state such thataggregating the cycles of the signal over time causes the oscillatingsignal to regularly alternate between the “high” and “low” (e.g.,on/off, or active/inactive) states. The term “duty cycle” may relate tothe fraction of a cycle of an oscillating signal associated with the“high” (e.g., on or active) state versus “low” (e.g., off or inactive)state. For example, a signal having a 50% duty cycle may be set to the“high” state for half of each cycle and substantially to the “low” statefor the complementary half of each cycle, accounting for latency andresponse times to transition between the high and low states.Aggregating multiple cycles over a time duration may result in anoscillating signal having a substantially uniform 50/50 distributionbetween the high/low states for the time duration. As another example, asignal having a 75% duty cycle may be set to the “high” state for threequarters of each cycle and to the “low” state substantially for thecomplementary one quarter of the cycle, accounting for latency andresponse times, such that aggregating multiple cycles over a timeduration results in an oscillating signal having a substantially uniform75/25 distribution between the high/low states for the time duration.For a visual display application, the high state may be associated witha high level of light output (e.g., illumination set to on or active andrelatively high-power consumption) and the low state may be associatedwith a low level of light output (e.g., illumination set to off orinactive and relatively low power consumption). Thus, controlling theduty cycle of a display signal may allow modifying the total lightoutput of the display signal. In some examples, reducing the duty cyclemay reduce the total light output, thereby reducing luminosity orintensity and power consumption, whereas and increasing the duty cyclemay increase the total light output, thereby increasing luminosity orintensity and power consumption. In some examples, reducing the dutycycle may reduce the opacity, whereas and increasing the duty cycle mayincrease the opacity. In some examples, for example when the frequencyof the cycles in the display signal is sufficiently high (e.g., above afusion threshold) and other steps are taken to maintain luminosity orintensity (for example by changing the maximum voltage), transitioningbetween the high and low states of each cycle may not be perceivable bythe human eye, allowing for a “smooth” visual user experience preventingeye strain. Thus, controlling the duty cycle for a visual display mayallow to smoothly transition between varying display configurations. Theterm “duty cycle control operations” may refer to one or more arithmeticand/or logical computations or procedures that may be performed by atleast one processor for controlling the duty cycle of a display signal.For example, the duty cycle control operations may include instructionsto implement pulse-width modulation (PWM), pulse-duration modulation(PDM), filters, signal compression or expansion, inversions, solutionsto differential equations, statistical and/or polynomial signalprocessing, stochastic signal processing, estimation and detectiontechniques and any additional signal processing techniques affecting theduty cycle of a display signal.

The term “wearable extended reality appliances” may refer to ahead-mounted device, for example, smart glasses, smart contact lens,headsets or any other device worn by a human for purposes of presentingan extended reality to the human, as described earlier. Thus, the atleast one processor may control the luminosity (e.g., brightness) and/orthe energy consumption for displaying content via a wearable extendedreality appliance by controlling the duty cycle of the display signal.

For example, during a first time duration, the at least one processormay perform a first PWM procedure to increase the duty cycle from 50% to75% to increase the brightness of content displayed via a wearableextended reality appliance. During a second time duration, the at leastone processor may perform a second PWM procedure to decrease the dutycycle from 75% to 50% to dim the display of content via the wearableextended reality appliance. As another example, the at least oneprocessor controlling the display of content via a wearable extendedreality appliance may perform a PWM procedure to display incomingmessages of a messaging application according to a 75% duty cycle, e.g.,to draw the attention of the user, and display a weather applicationaccording to a 50% duty cycle, e.g., as a background application.

By way of a non-limiting example, an exemplary implementation forperforming duty cycle control operations for wearable extended realityappliances is shown. FIG. 6 . Similar to FIG. 1 , FIG. 6 illustrates auser 100 wearing wearable extended reality appliance 110, with the noteddifference of an extended reality environment 620 including a firstdisplay region 602, a second display region 604, a physical wall 606,and a physical desktop 608. First display region 602 may be associatedwith the display of virtual screen 112 according to one duty cycleconfiguration (e.g., duty cycle configuration 610), and second displayregion 604 may be associated with the display of virtual widgets 114Cand 114D according to a different duty cycle configuration (e.g., dutycycle configuration 612). Processing device 460 (FIG. 4 ) may perform aPWM procedure (e.g., a duty cycle control operation) to control thedisplay of content via wearable extended reality appliance 110 such thatcontent associated with virtual screen 112 is displayed according toduty cycle configuration 610 (e.g., a 60%), for example to focus theattention of user 100 on virtual screen 112, while virtual widget 114Cmay be displayed according to duty cycle configuration 610 (e.g., 20%),for example, as a background application.

Some embodiments involve receiving data representing virtual content inan extended reality environment associated with a wearable extendedreality appliance. The term “receiving” may refer to accepting deliveryof, acquiring, retrieving, generating, obtaining or otherwise gainingaccess to. For example, information or data may be received in a mannerthat is detectable by or understandable to a processor. The data may bereceived via a communications channel, such as a wired channel (e.g.,cable, fiber) and/or wireless channel (e.g., radio, cellular, optical,IR). The data may be received as individual packets or as a continuousstream of data. The data may be received synchronously, e.g., byperiodically polling a memory buffer, queue or stack, or asynchronously,e.g., via an interrupt event. For example, the data may be received froman input device or sensor configured with input unit 202 (FIG. 1 ), frommobile communications device 206, from remote processing unit 208, orfrom any other local and/or remote source, and the data may be receivedby wearable extended reality appliance 110, mobile communications device206, remote processing unit 208, or any other local and/or remotecomputing device. In some examples, the data may be received from amemory unit, may be received from an external device, may be generatedbased on other information (for example, generated using a renderingalgorithm based on at least one of geometrical information, textureinformation or textual information), and so forth. The term “content”may refer to data or media. Such data or media may be formattedaccording to a distinct specification for presenting information to auser via an interface of an electronic device. For example, content mayinclude any combination of data formatted as text, image, audio, video,haptic, and any other data type for conveying information to a user. Theterm “virtual content” may refer to synthesized content that may existwholly within the context of one or more processing devices, for examplewithin an extended reality environment. Virtual content may thus bedistinguished from physical or real-world content that may exist or begenerated independent of a processing device. For example, voice datafor a synthesized digital avatar may be virtual content, whereas arecorded voice message of a human user may be associated with physical,real-world (e.g., non-virtual) content. By way of another example,virtual content may be a synthesized image, in contrast to a real-worldimage. The term “data representing virtual content” may include signalscarrying or encoding the virtual content. Such data (e.g., informationencoded into binary bits or n-ary qubits) may be formatted according toone or more distinct specifications to allow rendering virtual contentassociated with the data via a user interface of an electronic device.The term “extended reality environment”, e.g., also referred to as“extended reality”, “extended reality space”, or “extended environment”,may refer to all types of real-and-virtual combined environments andhuman-machine interactions at least partially generated by computertechnology, as described earlier. The extended reality environment maybe implemented via at least one processor and at least one extendedreality appliance (e.g., a wearable and/or non-wearable extended realityappliance). The term “associated with” may refer to the existence of arelationship, affiliation, correspondence, link or any other type ofconnection or correlation. The term “wearable extended realityappliance” may be understood as described earlier.

The wearable extended reality appliance may produce or generate anextended reality environment including representations of physical(e.g., real) objects and virtual content for viewing by the wearer. Forexample, wearable extended reality appliance may be a pair of smartglasses. The extended reality environment associated with the pair ofsmart glasses may include the field-of-view of the wearer of the smartglasses, e.g., a portion of the physical environment surrounding thewearer, as well as any virtual content superimposed thereon. Encodedinformation (e.g., data) for rendering (e.g., representing) virtualcontent may be obtained (e.g., received), for example, by a processorassociated with the pair of smart glasses. The encoded information maybe processed for displaying the virtual content within the extendedreality environment generated by (e.g., associated with) the pair ofsmart glasses (e.g., a wearable extended reality appliance). Forexample, the extended reality environment may include different displayregions, where received video content may be displayed in aforward-center region of the field-of-view of the wearer, and textcontent may be displayed as a notification (e.g., in a bottom rightcorner of the field-of-view of the wearer).

By way of a non-limiting example, FIG. 6 illustrates an extended realityenvironment 620 generated for user 100 by (e.g., at least) processingdevice 460 (FIG. 4 ) and wearable extended reality appliance 110.Processing device 460 may receive first data encoded as text content andsecond data encoded as image content for displaying to user 100 via(e.g., associated with) wearable extended reality appliance 110. Thetext content may represent values for virtual axes of a virtual bargraph displayed on virtual screen 112, and the image data may representvirtual widgets 114C and 114D.

Some embodiments involve identifying in the extended reality environmenta first display region and a second display region separated from thefirst display region. The term “identifying” may refer to recognizing,perceiving, or otherwise determining or establishing an association witha known entity, quantity, or value. The term “extended realityenvironment” may be understood as described earlier. The term “displayregion” may refer to a designated area or zone (e.g., physical and/orvirtual) inside the field-of-view of a wearer of a wearable extendedreality appliance. For example, the field of view of the wearer may bevisible via an electronic display screen (e.g., semi-transparent screen)of the wearable extended reality appliance. For example, the electronicdisplay may be an electroluminescent (EL), liquid crystal (LC), lightemitting diode (LED) include OLED and AMOLED, plasma, quantum dot, orcathode ray tube display, or any other type of electronic displaytechnology. The electronic display may include a region for presentingcontent (e.g., virtual content) together with (e.g., overlaid on,alongside, or otherwise co-presented with) the physical environmentsurrounding the wearer. For example, one part of the region may benon-transparent for presenting the virtual content, and another part ofthe region may be transparent for presenting the physical environment.The term “separate” may refer to detached, partitioned, or otherwisedisassociated, e.g., disjointed.

Thus, the extended reality environment may include multiple differentareas or zones for presenting content (e.g., display regions) that aredisassociated (e.g., separate) from each other. Each zone (e.g., displayregion) may include one or more transparent parts for viewing thephysical environment, and one or more non-transparent parts forpresenting virtual content overlaid on the presentation of the physicalenvironment. For example, a processing device may be configured torecognize or establish (e.g., identify) different display regionsaccording to one or more characteristics, such as relating to thewearer, the virtual content being displayed, the physical environment,software and/or hardware requirements of the wearable extended realityappliance, and/or any other characteristic relevant to the extendedreality environment.

For example, the different display regions may be recognized (e.g.,identified) according to the viewing angle of the wearer (e.g., thefront center may be the first display region, and the right side may bethe second display region). As another example, the different displayregions may be established (e.g., identified) according to differentattributes of the content displayed therein, such as the context (e.g.,high versus low priority, primary or peripheral content), type (e.g.,text, image, video), resolution (e.g., high versus low), representation(e.g., 2D versus 3D, grey scale versus color), or temporal attributes(e.g., current versus historical). In some embodiments, differentdisplay regions may be identified according to hardware characteristicsof one or more extended reality appliances used to implement theextended reality environment, such as the power consumption, resolutioncapability, channel capacity, memory requirements, or any other hardwarecharacteristic affecting the capability to render content. In someembodiments, different display regions may be identified according tocharacteristics of the user consuming content via the extended realityenvironment, such as the type of user (e.g., adult or child, young orold, disabled or able bodied), the user application (e.g., professionalor lay), the user activity (e.g., gaming, trading, viewing streamedcontent, editing a text document). In some embodiments, differentdisplay regions of the extended reality environment may be identifiedaccording to ambient conditions, such as lighting, temperature, thepresence of physical objects and/or background noise.

For example, a top-center area (e.g., a first display region) of theextended reality environment may be designated for rendering highpriority content (e.g., warnings or alerts), and a bottom-side region ofthe extended reality environment may be designated for rendering lowerpriority content (e.g., weather updates). As another example, aleft-oriented area may be designated for displaying 3D color images, anda right-oriented area may be designated for displaying white textagainst a dark background.

By way of a non-limiting example, turning to FIG. 6 , processing device460 (FIG. 4 ) may identify in extended reality environment 620 (e.g.,implemented via wearable extended reality appliance 110), a firstdisplay region 602 for displaying a first category of content, e.g.,relating to a work application for user 100, such as a bar chart andaccompanying text on virtual screen 112, and a second display region 604for displaying a second category of content, e.g., relating to personalapplications for user 100, such as virtual widget 114C associated withthe local weather forecast and virtual widget 114D associated withpersonal emails.

According to some embodiments, identifying the first display region andthe second display region is based on an analysis of the received data.For example, the received data may represent the virtual content. Theterms “identifying”, “display region”, and “received data” may beunderstood as described earlier. The term “based on” may refer toestablished or founded upon, or otherwise derived from. The term“analysis of the received data” may include examining or investigatingthe received data, such as by parsing one or more elements of thereceived data, and using the parsed elements to perform one or morecomputations, queries, comparisons, reasoning, deduction, extrapolation,interpolation, or any other logical or arithmetic operation, e.g., todetermine a fact, conclusion, or consequence associated with thereceived data. The analysis may be based on the data type, size, format,a time when the data was received and/or sent, communication and/orprocessing latency, a communications channel and/or network used toreceive the data, a source of the received data, the context under whichthe data was received, or any other criterion relevant to determining afact or consequence associated with the received data. For example, ananalysis of first received data may identify the first received data asvideo content for a live streaming application that may be displayed ina central region of the extended reality display environment, whereasanalysis of second received data may identify the second received dataas text content for an electronic mail application that may be displayedin a peripheral region of the extended reality display environment. Insome examples, a machine learning model may be trained using trainingexamples to identify display regions based on data representing virtualcontent. An example of such training example may include a sample datarepresenting sample virtual content, together with a label indicatingone or more desired display regions. The trained machine learning modelmay be used to analyze the received data representing the virtualcontent to identify the first display region and/or the second displayregion.

By way of a non-limiting example, turning to FIG. 6 , processing device460 (FIG. 3 ) may analyze first received data to determine that thefirst received data is graphic content. Based on this analysis,processing device 460 may identify first display region 602 forrendering the first received data, e.g., on virtual display 112. Inaddition, processing device 460 may analyze second received data todetermine a source of the second received data, e.g., a remote serverproviding weather updates. Based on this analysis, processing device 460may identify second display region 604 for rendering the second receiveddata, e.g., in association with virtual widget 114C.

According to some embodiments, identifying the first display region andthe second display region is based on an area of focus of a wearer ofthe wearable extended reality appliance. The terms “identifying”,“display region”, “based on”, and “wearable extended reality appliance”may be understood as described earlier. The term “wearer of the wearableextended reality appliance” may include a user donning, carrying, orotherwise being communicatively connected to the wearable extendedreality appliance, e.g., as clothing, an accessory (e.g., glasses,watch, hearing aid, ankle bracelet), a tattoo imprinted on the skin, asa sticker adhering to the surface of the skin, as an implant embeddedbeneath the skin (e.g., a monitor or regulator), or any other type ofwearable extended reality appliance. The term “area of focus” mayinclude a region or zone surrounding a point associated with a line ofsight (e.g., detected via an eye tracker), a head pose (e.g., angle,orientation, and/or inclination detected via an inertial measurementsensor), a gaze, or a region of an electronic display (e.g., a window,widget, document, image, application, or any other displayed element)selected for example via a keyboard, electronic pointing devicecontrolling a cursor, voice command, tracked gesture (e.g., head, handor any other type of gesture), eye tracking apparatus, or any othermeans for selecting a displayed element. The area of focus may include ashape (e.g., circle, square, ellipse, or any other geometric shape)surrounding or a document or application associated with a pointcorresponding to the line-of-sight of the wearer, or a position of thecursor.

Thus, the first and second display regions may be identified based onthe behavior of the user, e.g., where the wearer of the extended realityappliance is looking, pointing to, or otherwise indicating. For example,a forward center region of the virtual reality environment may beidentified as the first display region based on an eye tracker detectingthe line-of-sight of the wearer of the extended reality appliance. Asanother example, a bottom left region of the virtual reality environmentmay be identified as a second display region based on a selection via anelectronic pointing device.

By way of a non-limiting example, display region 602 in FIG. 6 may beidentified by processing device 460 (FIG. 4 ) based on an eye tracker(not shown) configured with wearable extended reality appliance 110detecting the line-of-sight of user 100 (e.g., a first area of focus ofuser 100). Concurrently, display region 604 may be identified byprocessing device 460 based on a selection of virtual widget 114C viaelectronic mouse 106 (e.g., a second area of focus of user 100).

According to some embodiments, identifying the first display region andthe second display region is based on characteristics of the extendedreality environment resulting from a physical environment of thewearable extended reality appliance. The terms “identifying”, “displayregion”, “based on”, “extended reality environment”, and “wearableextended reality appliance” may be understood as described earlier. Theterm “characteristics” may include attributes, properties, aspects,traits, or any other feature distinctly associated with the extendedreality environment. The term “physical environment” may refer to thereal-world surroundings of the wearable extended reality appliance, suchas the presence of walls, surfaces (e.g., floor, table tops, ceiling),obstructing objects (house plants, people, furniture, walls, doors),windows, and any other physical object potentially affecting the displayof content via the wearable extended reality appliance. The term“resulting from” may refer to following from, or consequent to. Thus,the extended reality environment may be affected by the physicalenvironment surrounding the wearable extended reality appliance, byincluding one or more objects facilitating the display of virtualcontent, (e.g., smooth, opaque, blank, white or pale colored, flat,and/or large surfaces) and/or one or more objects hampering the displayof virtual (e.g., obstructions, rough, small, dark, or transparentsurfaces, bright lights or other distracting objects). Accordingly, thefirst and second display regions may be identified according to one ormore characteristics of physical objects in the extended realityenvironment, such as the distance (e.g., far or close), color (e.g.,dark, light, varied or textured), size, texture (e.g., roughness,smoothness), opacity, transparency, shape, position (e.g., relative tothe head pose of the wearer), exposure to light or shadow, and any otherphysical characteristic affecting display capabilities. For example, ablank white wall facing the wearable extended reality appliance may beidentified as a first display region (e.g., requiring an average dutycycle to display content), and a window positioned adjacent to the wallmay be identified as a second display region (e.g., requiring a higherduty cycle to display content to overcome daylight). As another example,a desktop facing the wearable extended reality appliance may beidentified as a first display region, and a ceiling may be identified asa second display region.

By way of a non-limiting example, processing device 460 (FIG. 4 ) mayidentify wall 606 as display region 602 (FIG. 6 ) based on the positionrelative to user 100 (e.g., front-forward), its large size, white color,smooth texture, and the lack of a bright light source (e.g., window)hindering user 100 from seeing virtual content displayed thereon.Similarly, processing device 460 may identify desktop 608 of a desk asanother display region based on the position (e.g., front-down), basedon its flat, smooth surface, uniform color, and lack of an obstructingobject (e.g., a houseplant) hindering user 100 from seeing virtualcontent displayed thereon.

Some embodiments may involve receiving image data captured from thephysical environment of the wearable extended reality appliance using animage sensor included in the wearable extended reality appliance; andanalyzing the received image data to identify the first display regionand the second display region. The term “receiving”, “physicalenvironment”, “wearable extended reality appliance”, “display region”,and “identify” may be understood as described earlier. The term “imagedata” may refer to pixel data streams, digital images, digital videostreams, data derived from captured images, and data that may be used toconstruct one or more 2D and/or 3D images, a sequence of 2D and/or 3Dimages, 2D and/or 3D videos, or a virtual 2D and/or 3D representation,as described earlier. The image data may include data configured toconvey information associated with the visual characteristics of animage, for example as a graphic form or picture, as described earlier.For example, the image data may include at least one of pixels, voxelsor meta-data. The term “captured” may refer to the detection, sensing oracquisition of information using an optical sensor, such as by sensinglight waves reflecting off an object. The term “image sensor” mayinclude one or more sensory components capable of detecting andconverting optical signals in the near-infrared, infrared, visible, andultraviolet spectrums into electrical signals, as described earlier. Theelectric signals may be stored in memory and subsequently used toactivate the pixels of an electronic display to present the objectvisually. Examples of electronic image sensors may include digitalcameras, phone cameras, semiconductor Charge-Coupled Devices (CCDs),active pixel sensors in Complementary Metal-Oxide-Semiconductor (CMOS),or N-type metal-oxide-semiconductor (NMOS, Live MOS. The term “included”may refer to integrated or configured with. For example, an image sensormay be mechanically, optically, and/or electrically coupled to (e.g.,included with) the wearable extended reality appliance to sense physicalobjects, such as surfaces, light sources, obstacles, people, animals,and/or any other object present in or absent from the environment of thewearer of the wearable extended reality appliance. Thus, the wearableextended reality device may be provided with one or more image sensorsfor sensing the physical characteristics (e.g., objects, spaces, lightsources, shadows, and any other physical attribute) of the physicalenvironment surrounding the wearer of the wearable extended realityappliance. A processing device may convert the data sensed by the imagesensor to an image representing the physical environment.

The term “analyzing” may refer to investigating, scrutinizing and/orstudying a data set, for example, to determine a correlation,association, pattern or lack thereof within the data set or with respectto a different data set. The image data received by the image sensor maybe analyzed, for example using one or more image processing techniquessuch as convolutions, fast Fourier transforms, edge detection, patternrecognition, object detection algorithms, clustering, artificialintelligence, machine and/or deep learning, and any other imageprocessing technique, to identify the first and second regions. In someexamples, a machine learning model may be trained using trainingexamples to identify display regions based on images and/or videos. Anexample of such training example may include a sample image and/or asample video, together with a label indicating one or more desireddisplay regions. The trained machine learning model may be used toanalyze the received image data to identify the first display regionand/or the second display region. In some examples, at least part of theimage data may be analyzed to calculate a convolution of the at leastpart of the image data and thereby obtain a result value of thecalculated convolution. Further, in response to the result value of thecalculated convolution being a first value, one pair of regions may beidentified as the first display region and the second display region,and in response to the result value of the calculated convolution beinga second value, a different pair of regions may be identified as thefirst display region and the second display region. For example, imagedata sensed by the image sensor may be analyzed as described above.Based on the analysis of the image data, a vertical wall facing the usermay be identified as the first display region, and a horizontal tablesurface supporting an input device may be identified as the seconddisplay region.

By way of a non-limiting example with reference to FIG. 6 , wearableextended reality appliance 110 may be provided with a camera, such asimage sensor 472 (FIG. 4 ). Image sensor 472 may capture image data ofwall 606 facing user and desktop 608. Processing device 460 may receivethe image data from image sensor 472, e.g., via bus 400 and may analyzethe image data to identify wall 606 as the first region and desktop 608as the second region.

Some embodiments involve determining a first duty cycle configurationfor the first display region. The term “determining” may refer toestablishing or arriving at a conclusive outcome as a result of areasoned, learned, calculated or logical process. The term“configuration” may refer to a set up or an arrangement complying withone or more definitions or specifications. For example, a configurationmay include one or more settings assigning one or more values to one ormore parameters or variables to define a specific arrangement. The terms“duty cycle” and “display region” may be understood as describedearlier. Thus, a “duty cycle configuration” may include one or more setups, specifications or settings for one or more parameters affecting theduty cycle of a signal (e.g., a display signal), such as the ratio orpercent for each cycle during which the signal is set to “active” versus“inactive”, the frequency, amplitude and/or phase of the signal, theresponse time between the active” versus “inactive” states (e.g.,gradient), and/or any other attribute that may affect the duty cycle.Accordingly, a specific set of specifications or settings for the dutycycle (e.g., duty cycle configuration) may be established for the firstdisplay region. For example, if the first display region is associatedwith high priority content, and/or exposed to a strong light source(e.g., a window exposing daylight), the duty cycle configuration may bedetermined to cause a more intense display (e.g., by setting a higherluminosity). Conversely, if the first display region is associated withlow priority content or positioned in a relatively dark region of theextended reality environment, the duty cycle configuration may bedetermined to cause a dimmer display (e.g., by setting a lowerluminosity). As another example, the first display region may beassociated with a default duty cycle defined in advance, such as aprimary display region automatically associated with a high (e.g., 60%)duty cycle.

By way of a non-limiting example, turning to FIG. 6 , processing device460 (FIG. 4 ) may determine a duty cycle configuration 610 for displaysignals associated with display region 602. Duty cycle configuration 610may include one or more settings causing content displayed via projector454 in display region 602 to correspond to a 60% duty cycle, such thatslightly more than half (e.g., approximately 60%) of every cycle of thedisplay signal is set to “high” or “active”, and slightly less than half(e.g., approximately 40%) of every cycle is set to “low” or “inactive”.

Some embodiments involve determining a second duty cycle configurationfor the second display region, wherein the second duty cycleconfiguration differs from the first duty cycle configuration. The terms“determining”, “duty cycle”, “configuration”, and “display region” maybe understood as described earlier. The term “differs” may refer tobeing distinguished or distinct from, or otherwise being dissimilar.Thus, a second duty cycle configuration determined for the seconddisplay region may be dissimilar to (e.g., distinct from) the first dutycycle configuration determined for the first display region such thatthey are not the same. For example, the first duty cycle configurationmay cause content to be displayed according to a 30% duty cycle, e.g.,relatively dim and drawing little power, and the second duty cycleconfiguration may cause content to be displayed according to an 80% dutycycle, e.g., relatively bright and drawing considerably more power.

By way of a non-limiting example, turning to FIG. 6 , processing device460 (FIG. 4 ) may determine a duty cycle configuration 612 for displaysignals associated with display region 604 that differs from duty cycleconfiguration 610 determined for display region 602. Duty cycleconfiguration 612 may include one or more settings causing contentdisplayed via projector 454 in display region 604 to correspond to a 20%duty cycle, such that only a small fraction (e.g., approximately 20%) ofevery cycle of the display signal is set to “high” or “active”, and apredominant portion of the signal (e.g., approximately 80%) of everycycle is set to “low” or “inactive”. Consequently, content may bedisplayed in display region 602 differently than content displayed indisplay region 604.

Some embodiments involve causing the wearable extended reality applianceto display the virtual content in accordance with the determined firstduty cycle configuration for the first display region and the determinedsecond duty cycle configuration for the second display region. The term“causing” may include triggering, inducing or taking an action to bringabout a particular consequence or deterministic outcome. The term“display” may refer to presenting visually, for example by controllingthe activation of one or more pixels of an electronic display tovisually exhibit content. For example, some regions of an extendedreality display may include pixels activated by circuitry for displayingvirtual content overlaid on non-activated (e.g., transparent) regions ofthe display presenting the real world. The terms “extended realityappliance”, “virtual content”, “duty cycle configuration” (e.g.,determined duty cycle configuration), and “display region” may beunderstood as described earlier. Thus, after determining the first andsecond duty cycle configurations the wearable extended reality appliancemay be caused to display content in each of the first and second displayregions according to the first and second duty cycle configurations,respectively.

For example, a processing device executing software instructions maycontrol hardware circuitry (e.g., switches, diodes, transistors,controllers, filters, samplers, converters, compressors) and/or softwareequivalents to display content via the wearable extended realityappliance. The processing device may thus direct different displaysignals to different display regions of the extended reality environmentto cause specific content to be displayed in certain regions. Forexample, display signals may be directed to display content in differentregions according to one or more criterion as described earlier, such asthe content type, context, priority, the physical environment, ambientconditions (e.g., lighting, noise, distractions), and/or any othercriterion relevant to the display of content. In addition, theprocessing device may modify the display signals targeted for eachdisplay region using one or more signal processing techniques (e.g.,analog and/or digital, linear and/or non-linear, discrete and/orcontinuous time) to affect the duty cycle of the display signals. Signalprocessing techniques affecting the duty cycle may include, for example,filters, transforms, modulations (e.g., PWM or PDM), inversions,differential equations, statistical and/or polynomial signal processing,stochastic signal processing, estimation and detection techniques, andany other signal processing technique affecting the duty cycle. Theprocessing device may control aspects or parameters affecting the dutycycle such as the frequency and/or amplitude of the display signal, thepercent during which each cycle of the display signal is set to “active”versus “inactive”, the latency and/or responsiveness of the switchingbetween the “active” and “inactive states within each cycle (e.g.,expressed as a time delay within each cycle or gradient to transitionbetween the active and inactive states), and/or any other factoraffecting the duty cycle of the display signals.

The first and second duty cycle configurations may be applied to thedisplay signals based on time, space, context or association, frequencyof use, head pose, background noise, physical and/or virtualdistractions, and/or any other criterion. Applying the first/second dutycycles based on time may include, for example, applying the first dutycycle configuration during morning hours and the second duty cycleconfiguration in the evening. Applying the first/second duty cyclesbased on space may include, for example applying the first duty cycleconfiguration for displaying content against a wall and the second dutycycle configuration for displaying on the surface of a desk. Applyingthe first/second duty cycles based on context or association mayinclude, for example applying the first duty cycle configuration forhighly relevant or important content and the second duty cycleconfiguration for less relevant content. Applying the first/second dutycycles based on frequency of use may include, for example, applying thefirst duty cycle configuration for frequently used virtual widgets oraccessories and the second duty cycle configuration for infrequentlyused widgets or accessories. Applying the first/second duty cycles basedon head pose may include, for example, applying the first duty cycleconfiguration when the head of the user faces forward and the secondduty cycle configuration when the user turns his head sideways. Applyingthe first/second duty cycles based on background noise may include, forexample, increasing the duty cycle in response to detecting distractingsounds, and decreasing the duty cycle in the absence of backgroundnoise. Applying the first/second duty cycles based on physical and/orvirtual distractions may include, for example, increasing the duty cyclein response to detecting a person, animal or virtual avatar entering theextended reality environment.

Additionally, or alternatively, the first and second duty cycleconfigurations may be applied to the same object at different times, todifferent objects displayed simultaneously, to different regions of theextended reality environment, according to context or association andany other criterion differentiating displays of content via the wearableextended reality appliance. Applying the first/second duty cycles to thesame object at different times may be based on, for example, the time ofday and/or the frequency that the object is being used, changes toambient illumination, or a changing head pose or posture of the user.Applying the first/second duty cycles to different objects displayedsimultaneously may include, for example, displaying video content usinga first duty cycle configuration simultaneously with displaying avirtual widget using a second duty cycle configuration. Applying thefirst/second duty cycles to different regions may include, for example,displaying objects docked to a desktop using a first duty cycleconfiguration and objects docked to a wall using a second duty cycleconfiguration. Applying the first/second duty cycles according tocontext or association may include, for example, displaying moreimportant content according to a first duty cycle configuration, andless relevant content according to the second duty cycle configuration.

By way of a non-limiting example, processing device 460 (FIG. 4 ) maycause projector 454 to display content within display region 602 (FIG. 6) according to duty cycle 610 (e.g., 60%) and content within displayregion 604 according to duty cycle 612 (e.g., 20%). For example,processing device 460 may determine that display region 604 is situatedin a relatively dark region within the physical space surrounding user100, and that a duty cycle of 20% is therefore sufficient. Additionally,or alternatively, processing device 460 may determine that displayregion 602 corresponds to high priority content, and that a duty cycleof 60% may better draw the attention of user 100.

Some embodiments may involve determining a spatial distribution of thevirtual content in the extended reality environment, and wherein atleast one of the first duty cycle configuration and the second dutycycle configuration is determined based on the spatial distribution ofthe virtual content. The terms “determining”, “virtual content”,“extended reality environment”, “duty cycle configuration”, and “basedon”, may be understood as described earlier. The term “spatialdistribution” may refer to an arrangement, layout or allocation of thevirtual content in the extended reality environment, such as where inthe extended reality environment content is displayed at a given momentin time. For example, a spatial distribution may be determined based ona density threshold for displaying content (e.g., to avoid a cluttereddisplay), the field of view of the user, ambient lighting conditions,the presence of physical and/or virtual objects, and any othercriterion. Determining the first/second duty cycle configuration for aspatial distribution based on a density threshold may include, forexample, reducing the duty cycle when the density of the displayedcontent exceeds the threshold, e.g., to prevent eye strain. Determiningthe first/second duty cycle configuration for a spatial distributionbased on the field of view of the user may include, for example,displaying content in the periphery according to a lower duty cycleconfiguration, and content in the center according to a higher dutycycle configuration, e.g., to facilitate concentration. Determining thefirst/second duty cycle configuration for a spatial distribution basedon ambient lighting conditions may include, for example, applying ahigher duty cycle configuration for content displayed in brightly litareas and a lower duty cycle configuration for content displayed indimly lit, or shadowed areas. Determining the first/second duty cycleconfiguration for a spatial distribution based on the presence ofphysical and/or virtual objects may include, for example, increasingand/or lowering the duty cycle when content is displayed in proximity tocertain objects (e.g., based on the object type, color, size, lightreflectance, light absorbance, or any other visible criterion).

As another example, a spatial distribution may cause a virtual keyboardto be fixed (e.g., docked) to a physical desktop (e.g., regardless ofthe head pose of the user) while causing a virtual screen to follow theuser's gaze, e.g., anywhere within the extended reality environment. Ahigher duty cycle configuration may be applied to display the fixedvirtual keyboard, and a lower duty cycle may be applied to display thevirtual screen following the user's gaze (e.g., to prevent motionsickness). Another spatial distribution may redistribute, resize, orcollapse a plurality of virtual widgets into a list when the number ofvirtual widgets exceeds a threshold. In such a case, the duty cycle fordisplaying the virtual widgets may be lowered.

By way of a non-limiting example, processing device 460 (FIG. 4 ) maydetermine to spatially distribute virtual screen 112 in display region602 of FIG. 6 , e.g., in the direct line of sight of user 100 when user100 is facing wall 606, and virtual widgets 114C and 114D in aperipheral region of the field of view of user 100. Based on thisspatial distribution, processing device 460 may determine to displaycontent in virtual screen 112 according to duty cycle configuration 610,and virtual widgets 114C and 114C according to duty cycle configuration610.

Some embodiments may further involve detecting a head motion of a wearerof the wearable extended reality appliance, and wherein at least one ofthe first duty cycle configuration and the second duty cycleconfiguration is determined based on the detected head motion of thewearer. The term “detecting” may include discovering, noticing orascertaining, for example in response to sensing or otherwise becomingaware of something. For example, a sensor (e.g., optical, electricand/or magnetic, acoustic, motion, vibration, heat, pressure, olfactory,gas, or any other type of sensor) may sense a signal that may beanalyzed to discover or ascertain, and thereby detect a physicalphenomenon, such as a head motion. The term “head motion” may refer toany movement, for example enabled by the neck and/or shoulder muscles,which changes the position of the head (e.g., the part of the body fromthe neck upwards, including the ears, brain, forehead, cheeks, chin,eyes, nose, and mouth). Examples of head motion may include tilting(e.g., up and down motion), rotating (e.g., left or right motion),leaning (e.g., sideways motion), shifting (e.g., 360 degrees parallel tothe floor plane, as a result of moving at least the upper body), and anycombination thereof. The terms “wearer of the wearable extended realityappliance”, “duty cycle configuration”, “determined”, and “based on” maybe understood as described earlier. For example, the head motion of thewearer of the wearable extended reality appliance may be detected withrespect to the body of the wearer, a stationary physical object in thevicinity of the wearer, a virtual object displayed via the extendedreality appliance, and/or any combination thereof. At least one motionsensor may be provided to track any of the position, orientation, pose,and/or angle of the head of the wearer to detect a head motion. Examplesof motion sensors that may be used include an IMU sensor (e.g.,including one or more of a gyroscope, compass, and accelerometer), acamera (e.g., optic, IR), an acoustic sensor (e.g., sonar, ultrasound),an RFID sensor, and any other sensor configured to sense motion. Forexample, a combination of an IMU sensor integrated with the wearableextended reality appliance and an optical detector (e.g., camera)positioned to detect the head of the wearer may operate together totrack head motions of the wearer, such as up/down, left/right, tilt,rotation, translation (e.g., due to walking), and any other type of headmotion. Data collected by the at least one motion sensor may be receivedand analyzed by a processor to track the head of the wearer over time.

Thus, at least one of the duty cycle configurations may be determinedbased on the head motion of the wearer of the wearable extended realityappliance. For example, when the wearer is facing a virtual screen, avirtual widget may be displayed according to a first (e.g., high) dutycycle configuration, e.g., to facilitate the interfacing of the wearerwith the virtual widget. However, when the wearer turns his head awayfrom the virtual screen, for example to take a rest break, the virtualwidget may be displayed according to a second (e.g., lower) duty cycleconfiguration, to prevent motion sickness or distractions during therest break, while still allowing the wearer to interface with thevirtual widget if necessary. As another example, two virtual screens maybe displayed to the wearer simultaneously, where the display of eachvirtual screen may toggle between the first and second duty cycleconfigurations depending on the head orientation of the wearer of theextended reality appliance. For example, the configuration with thehigher duty cycle may be used to display whichever of the two virtualscreens is currently in a direct line of sight of the wearer, and theconfiguration with the lower duty cycle may be used for the othervirtual screen (e.g., not in the direct line of sight). As the wearermoves his direct line of sight to the other of the two virtual screens,the duty cycle configuration may be switched.

By way of a non-limiting example, turning to FIG. 7 , an exemplaryimplementation for basing the duty cycle on a determined head motion isshown. FIG. 7 is substantially similar to FIG. 6 with the notabledifference of a display region 702 positioned against wall 606 at eyelevel with user 100, and display region 706 positioned on desktop 608.Virtual content on virtual screen 112 may be displayed in display region702, and a virtual widget 710 may be displayed in display region 706.Two duty cycle configurations 704 (e.g., 60%) and 708 (e.g., 20%) may beprovided to follow the gaze of user. When the gaze of user is directedtowards display region 702, virtual screen 112 may displayed accordingto duty cycle 704, e.g., to provide a more intense display, whereasvirtual widget 710 may be displayed according to duty cycle 708, e.g.,to conserve energy. When the head of user 100 tilts downwards, away fromdisplay region 702 and towards display region 706, processing device 460(FIG. 4 ) may detect the head motion (e.g., in conjunction with motionsensor 473) and switch the duty cycle configurations. Thus, for example,when the head of user 100 tilts downwards, virtual screen 112 may bedisplayed according to duty cycle configuration 708 (e.g., to conserveenergy) and virtual widget 710 may be displayed according to duty cycleconfiguration 704, (e.g., to provide a more intense display). When user100 tilts head 714 upwards once more to face virtual screen 112,processing device 460 may switch duty cycle configurations 704 and 708again, accordingly.

Some embodiments may provide a non-transitory computer readable mediumcontaining instructions that when executed by at least one processorcause the at least one processor to perform duty cycle controloperations for wearable extended reality appliances, the operationscomprising: receiving data representing virtual content in an extendedreality environment associated with a wearable extended realityappliance; detecting in the extended reality environment a first headmotion and a second head motion of a wearer of the wearable extendedreality appliance; determining a first duty cycle configuration based onthe first head motion; determining a second duty cycle configurationbased on the second head motion, wherein the second duty cycleconfiguration differs from the first duty cycle configuration; andcausing the wearable extended reality appliance to display the virtualcontent in accordance with the first duty cycle configuration upondetecting the first head motion and in accordance with the second dutycycle configuration upon detecting the second head motion. The terms“non-transitory computer-readable medium”, “instructions”, “processor”,“duty cycle control operations”, “wearable extended reality appliances”,“receiving”, “data representing virtual content”, “extended realityenvironment”, “associated with”, “detecting”, “head motion”, “wearer ofthe wearable extended reality appliance”, “determining”, “duty cycleconfiguration”, “based on”, “differs”, “causing”, “display”, and“virtual content” may be understood as described earlier. Thus,according to some embodiments, instead of using different displayregions, different head motions of the wearer of the wearable extendedreality appliance may be used to determine different (e.g., first andsecond) duty cycle configurations and to cause virtual content to bedisplayed according to different duty cycle configurations. In someexamples, data captured using inertial sensors, accelerometers,gyroscopes and/or magnetometers included in the wearable extendedreality appliance may be analyzed to determine head motion (such as thefirst head motion and/or the second head motion), head position and/orhead direction. In some examples, image data captured using imagesensors included in the wearable extended reality appliance may beanalyzed (for example, using egomotion algorithms, using ego-positioningalgorithms, etc.) to determine head motion (such as the first headmotion and/or the second head motion), head position and/or headdirection.

For example, while stationed at a physical work station (e.g., seated ina chair facing a physical wall) a wearer of a wearable extended realityappliance may turn his head level to face the wall. The motion (e.g.,the first head motion) may be detected and used to determine a firstduty cycle configuration for displaying virtual content (e.g., text on avirtual screen). For example, the first duty cycle configuration may berelatively high to allow the wearer to interface with the displayedvirtual content. When the wearer turns his head away from the wall, forexample, due to a distraction, the head turning motion may be detectedas a second head motion. The second head motion may be used to determinea second (e.g., lower) duty cycle configuration for displaying thevirtual content, previously displayed according to the first (e.g.,higher) duty cycle configuration, for example to conserve energy sincethe focus of the user is no longer on the wall. As another example, afirst head motion (e.g., gesture) by the wearer of the wearable extendedreality appliance may be associated with invoking a messaging widget.Upon detecting the first head motion, the messaging widget may bedisplayed according to a first (e.g., relatively high) duty cycleconfiguration. When the wearer turns his head down to focus on thedesktop, the downward motion (e.g., the second head motion) may bedetected and used to determine a second (e.g., lower) duty cycleconfiguration. The messaging widget may be continually displayed by theextended reality appliance to follow the gaze of the wearer. However,the messaging widget may now be displayed according to the second (e.g.,lower) duty cycle configuration instead of the first (e.g., higher) dutycycle configuration, for example to prevent nausea.

By way of a non-limiting example, FIG. 8 illustrates an exemplaryimplementation for using head motions to determine the duty cycleconfigurations, in place of display regions. FIG. 8 is substantiallysimilar to FIG. 7 with the noted difference that the duty cycleconfiguration may be based on a head motion, independent of the displayregion. Thus, a detected head motion may be used determine the dutycycle configuration to apply, instead of the display region.Accordingly, a first head motion leading to head position 802 may beassociated with a duty cycle configuration 804, and a second head motionleading to head pose 806 may be associated with a second duty cycleconfiguration 808, e.g., instead or in place of first and second displayregions associated with first and second duty cycle configurations 804and 808, respectively. For example, first head motion leading to headposition 802 may correspond to user 100 invoking virtual screen 112, andsecond head motion leading to head pose 806 may correspond to user 100turning the focus away from virtual screen. Processing device 460 inconjunction with motion sensor 473 (FIG. 4 ) may detect the first headmotion (e.g., leading to first head position 802) and determine aninvocation of virtual screen 112. In response, processing device 460 mayapply duty cycle configuration 804 to display virtual screen 112. Forexample, duty cycle configuration 802 may be relatively high (e.g., 60%)to allow user to see virtual content on virtual screen 112. Processingdevice 460 in conjunction with motion sensor 473 may detect the secondhead motion (e.g., positioning the head of user 100 into second headpose 806) and determine the focus turned away from virtual screen 112.In response, processing device 460 may apply duty cycle configuration808 for displaying virtual screen 112. For example, duty cycleconfiguration 808 may be relatively low (e.g., 20%) to conserve energysince user 100 is no longer focused on virtual screen 112.

According to some embodiments, the at least one of the first duty cycleconfiguration and the second duty cycle configuration is determinedbased on a speed associated with the detected head motion of the wearer.The term “speed” may refer to velocity, pace, or rate of an activity.Thus, the rate at which the wearer moves his head may be used todetermine the first and/or second duty cycle configurations, and/orwhich duty cycle configuration to apply for displaying virtual content.In some embodiments, the speed that the wearer moves his head may becompared to a predefined threshold, such that if the speed is greaterthan the threshold, one of the duty cycle configurations may be applied,and if the speed is less than the threshold, the other duty cycleconfiguration may be applied. For example, the threshold may beassociated with nausea or motion sickness. When speed of the head motionis below the threshold, a first duty cycle configuration (e.g., higherof the two duty cycle configurations) may be applied, e.g., to enhancethe display of the virtual content. However, when the wearer moves hishead at a speed exceeding the threshold, a second duty cycleconfiguration (e.g., lower of the two duty cycle configurations) may beapplied, e.g., to prevent nausea. For example, when the wearer iswalking slowly, e.g., the head motion is below the threshold, the higherof the two duty cycle configurations may be applied, and when the weareris walking quickly, e.g., the head motion exceeds the threshold, thelower of the two duty cycle configurations may be application, e.g., toprevent motion sickness. As another example, a predefined head gesture,e.g., performed at a particular speed, may be associated with invoking aspecific application. When the detected head motion corresponds topredefined head gesture at the particular speed (e.g., the wearerdeliberately moved his head to invoke the application), the first dutycycle configuration may be applied to display the invocation of thespecific application. However, when the speed of the detected headmotion does not correspond to the particular speed for the predefinedhead gesture (e.g., the wearer moved his head arbitrarily, with nointention of invoking the application), the second duty cycleconfiguration may be applied to display virtual content, e.g., toprevent motion sickness.

By way of a non-limiting example, turning to FIG. 8 , a smooth andsteady downwards motion of the head from head pose 802 to 806 may beassociated with displaying a virtual widget 810 on desktop 608. Upondetecting a smooth head motion by user 100 moving the head from headpose 802 to head pose 806, processing device 460 in conjunction withmotion sensor 473 (FIG. 4 ) may determine that user 100 has deliberatelymoved his head to perform a predefined gesture associated withdisplaying virtual widget 810 on desktop 608. In response, processor 460may display virtual widget 810 on surface 608 according to first dutycycle configuration 806, e.g., 60% duty cycle for an enhanced display.However, upon detecting an abrupt head motion by user 100 from head pose802 to 806 (e.g., a head motion differing from the predefined gesturedue to the speed), processing device 460 in conjunction with motionsensor may determine that the detected head motion is associated withsomething other than displaying virtual widget 810, e.g., a typing onkeyboard 104. In response, processor 460 may avoid displaying virtualwidget 810, and instead change the display of virtual content on virtualscreen from duty cycle configuration 804 (e.g., higher intensity) toduty cycle configuration 808 (e.g., lower intensity), e.g., to conserveenergy.

According to some embodiments, the at least one of the first duty cycleconfiguration and the second duty cycle configuration is determinedbased on a direction associated with the detected head motion of thewearer. The term “direction” may refer to an orientation, course, orpath along which something moves. Thus, the orientation of a user's heador a path along which the user moves his head may be used to determinethe first and/or second duty cycle configurations and/or which dutycycle configuration to apply for displaying virtual content. Forexample, a gesture recognition application may associate a right turn ofthe head with invoking an application and a left turn of the head withpausing an application. In response to detecting a right turn of thehead, the first duty cycle configuration may be applied to displayvirtual content for the invoked application. In response to detecting aleft turn of the head, the second duty cycle configuration may beapplied to display virtual content for the paused application. Asanother example, a head tilt downwards up to a predefined threshold maybe associated with a distraction of fatigue, and thus the lower of thetwo duty cycle configurations may be applied, whereas a head tiltdownwards beyond the predefined threshold may be associated with adeliberate gesture to invoke an application, such as to display avirtual widget on a desktop, and thus the higher of the two duty cycleconfigurations may be applied.

By way of a non-limiting example, turning to FIG. 8 , head pose 806(e.g., downwards) may be defined as a predefined threshold for invokingvirtual widget 810. During a first head motion by user 100 tilting thehead downwards from head pose 802, but stopping before reaching headpose 806, processing device 460 in conjunction with motion sensor 473(FIG. 4 ) may detect the first head motion and determine that user 100is fatigued. In response, processing device 460 may display virtualcontent via virtual screen 112 according to duty cycle configuration 808(e.g., 20%). During a second head motion by user 100 tilting the headdownwards to reach head pose 806, processing device 460 in conjunctionwith motion sensor 473 (FIG. 4 ) may detect the second head motion anddetermine that user 100 wishes to invoke virtual widget 810. Inresponse, processing device 460 may display virtual widget 810 ondesktop 608 according to duty cycle configuration 804 (e.g., 60%).

Some embodiments may further involve determining an area of focus of awearer of the wearable extended reality appliance, and wherein at leastone of the first duty cycle configuration and the second duty cycleconfiguration is determined based on the determined area of focus. Theterm “area of focus” may be understood as described earlier. Thus, thearea of the extended reality environment that the wearer is currentlylooking at or otherwise focused on may be used to determine the firstand/or second duty cycle configurations and/or which duty cycleconfiguration to apply for displaying virtual content. For example, ifthe area of focus is inside a predefined region of the extended realityenvironment, the first duty cycle configuration may be applied, and ifthe area of focus is outside the predefined region, the second dutycycle configuration may be applied. In some examples, the area of focusof the wearer of the wearable extended reality appliance may bedetermined based on a gaze direction of the wearer, and the gazedirection may be determined based on an analysis (for example, using agaze detection algorithm) of one or more images of one or two of thewearer's eyes. For example, the one or more images may be captured usingan image sensor included in the wearable extended reality appliance. Insome examples, the area of focus of the wearer of the wearable extendedreality appliance may be determined based on a head direction of thewearer, and the head direction may be determined as described above. Insome examples, the area of focus of the wearer of the wearable extendedreality appliance may be determined based on an interaction of thewearer with an object (such as a virtual object or a physical object) inthat area, for example through gestures, through a pointing device,through a keyboard, and any other user interfacing technique.

By way of a non-limiting example, turning to FIG. 7 , when processingdevice 460 (FIG. 4 ) detects the focus of user 100 on virtual screen112, content may be displayed on virtual screen 112 according to dutycycle 704 (e.g., 60%). When processing device 460 detects the focus ofuser 100 away from virtual screen 112 (e.g., towards virtual widget 710,content may be displayed on virtual screen 112 according to duty cycle708 (e.g., 20%).

Some embodiments may further involve detecting a physical object locatedin proximity to the wearable extended reality appliance, and wherein atleast one of the first duty cycle configuration and the second dutycycle configuration is determined based on the detected physical object.The terms “detecting”, “wearable extended reality appliance”, and “dutycycle configuration” may be understood as described earlier. Regardingdetecting a physical object, for example, a sensor (e.g., electricand/or magnetic, optic, acoustic, vibration, olfactory, and any othertype of physical sensor) may detect one or more physical characteristicsof an object based on a signal emitted from, reflected off, or absorbedby the object. Examples of physical characteristic of a physical objectthat may be detected may include a distance and/or orientation relativeto the wearable extended reality appliance, a color, texture, size(e.g., minimal size), optical properties (e.g., glossiness, roughness,reflectance, fluorescence, refractive index, dispersion, absorption,scattering, turbidity, and any other optical property). For example,image data captured using an image sensor included in the wearableextended reality appliance may be analyzed using an object detectionalgorithm to detect the physical object. The term “physical object” mayinclude a real or tangible item, such as may be governed by classicallaws of physics. The term “located” may refer to a position, placementor station, e.g., in a physical environment. The term “proximity” mayrefer to being adjacent, or close to (e.g., within a predefineddistance). Thus, characteristics of physical objects, such as size,optical properties, distance and/or orientation from the wearableextended reality appliance may be used to determine the first and/orsecond duty cycle configurations and/or which duty cycle configurationto apply for displaying virtual content. For example, virtual contentdisplayed next to a brightly colored physical object positioned inproximity to the wearable extended reality appliance may be displayedusing a relatively high duty cycle configuration, e.g., to allowdistinguishing the virtual content next to the brightly colored physicalobject. Similarly, virtual content displayed next to a small and/or dullobject may be displayed using a relatively low duty cycle configuration,e.g., to allow distinguishing the small, dull object next to the virtualcontent.

In some examples, when the physical object is a person approaching thewearer of the wearable extended reality appliance, one value may beselected for the first duty cycle configuration, and when the physicalobject is a person not approaching the wearer of the wearable extendedreality appliance, a different value may be selected for the first dutycycle configuration. For example, tracking algorithms may be used toanalyze images of the person and determine a trajectory of the person,and the determined trajectory may be analyzed to determine if the personis approaching the wearer. In some examples, when the physical object isa person interacting with a wearer of the wearable extended realityappliance, one value may be selected for the first duty cycleconfiguration, and when the physical object is a person not interactingwith a wearer of the wearable extended reality appliance, a differentvalue may be selected for the first duty cycle configuration. Forexample, audio data captured using an audio sensor included in thewearable extended reality appliance may be analyzed (for example, usinga speech recognition algorithm) to determine whether the person isverbally interacting with the wearer. In another example, image datacaptured using an image sensor included in the wearable extended realityappliance may be analyzed (for example, using a gesture recognitionalgorithm) to determine whether the person is interacting with thewearer through gestures.

By way of a non-limiting example, processing device 460 (FIG. 4 ) mayapply duty cycle configuration 610 (e.g., 60%) (see FIG. 6 ) to displayvirtual screen 112 based on the relatively close proximity to smartglasses 110. Similarly, processing device 460 may apply duty cycleconfiguration 612 (e.g., 20%) to display virtual content on the far edgeof desktop 608 based on the relatively far distance from smart glasses110.

Some embodiments may further involve detecting a virtual object in theextended reality environment, and wherein at least one of the first dutycycle configuration and the second duty cycle configuration isdetermined based on the detected virtual object. The term “virtualobject” may refer to a visual presentation rendered by a computer in aconfined region and configured to represent an object of a particulartype (such as an inanimate virtual object, an animate virtual object,virtual furniture, a virtual decorative object, virtual widget, or othervirtual representation) as described earlier. The terms “detecting”,“extended reality environment”, “duty cycle configuration”, “determined”may be understood as described above. For example, a virtual object maybe detected by a processor controlling the display of content via thewearable extended reality appliance, such as by detecting an increase inmemory and/or bandwidth consumption (e.g., indicating a video played ina picture-in-picture, the introduction of a virtual avatar), detectingthe response of the wearer to the display of the virtual object (e.g.,via an eye tracker, an event listener configured with an electronicpointing device, a voice recognition application configured with amicrophone), and any other method for detecting virtual content.

Thus, the existence and/or the display characteristics of a virtualobject in the extended reality environment may be used to determine thefirst and/or second duty cycle configurations and/or which duty cycleconfiguration to apply. For example, the duty cycle configuration may bebased on a distance and/or orientation of the virtual object relative tothe wearable extended reality appliance, on the size of the virtualobject, on an optical property of the virtual object (e.g., color,luminance, opacity, pixel intensity), and any other visual property ofthe virtual object. For example, a high duty cycle configuration may beapplied to display content in proximity to a large or bright virtualobject, and a lower duty cycle configuration may be applied to displaycontent in proximity to a dim or translucent virtual object.

By way of a non-limiting example with reference to FIG. 6 , upondetecting the start of a streamed video playing in a picture-in-picture622 (e.g., based on increase in memory usage), processing device 460 maydetermine to display virtual screen 112 according to duty cycleconfiguration 612 instead of duty cycle configuration 610, e.g., toconserve resources.

Some embodiments may further involve detecting a physical movement inproximity to the wearable extended reality appliance, and wherein atleast one of the first duty cycle configuration and the second dutycycle configuration is determined based on the detected physicalmovement. The term “physical movement” may refer to an activity ormotion in the real physical world, e.g., requiring an expenditure ofenergy. For example, an object falling, a person, animal, or robotpassing by, may be physical movements. The terms “detecting”,“proximity”, “wearable extended reality appliance”, and “duty cycleconfiguration” may be understood as described above. Thus, physicalmovement may be detected based on speed, the region and/or proportionthat the physical movement occupies in the extended reality environment,the type of movement (e.g., sudden versus slow), the entity performingthe movement (e.g., virtual or real, human or inanimate). The physicalmovement may be detected via one or more detectors configured in theproximity to the wearable extended reality appliance, such as anoptical, acoustic, radio or any other type of detector. For example, amotion detection, visual activity or event detection algorithm may beapplied to a sequence of images (e.g., video) captured via a camera.Thus, the existence of physical movement in proximity to the wearableextended reality appliance may be used to determine the first and/orsecond duty cycle configurations and/or which duty cycle configurationto apply for displaying virtual content. For example, a child enteringthe extended reality environment or an object falling may cause virtualcontent to be displayed according to a lower duty cycle configuration,e.g., to draw the attention of wearer away from the extended realityenvironment so that the wearer may be aware of the child or the fallingobject.

By way of a non-limiting example with reference to FIG. 6 , a ball 624may be tossed by a child in proximity to user 100 while user 100 isworking at home via wearable extended reality appliance 110. A camera626 positioned on wall 606 in proximity to smart glasses 110 may capturea video of the motion of ball 624 and provide the video to processingdevice 460 (FIG. 4 ). Processing device 460 may analyze the video anddetect the physical movement of ball 624. In response, processing device460 may cause virtual content, previously displayed via wearableextended reality appliance 110 according to duty cycle 610, to bedisplayed according to duty cycle configuration 612, e.g., as a way ofnotifying user 100 of the presence of ball 624.

Some embodiments may involve identifying a type of virtual contentincluded in the first display region, and wherein at least one of thefirst duty cycle configuration and the second duty cycle configurationis determined based on the type of virtual content included in the firstdisplay region. The term “type of virtual content” may refer to acontext, classification, genre, or any other category of virtualcontent. The terms “virtual content”, “display region”, and “duty cycleconfiguration” may be understood as described earlier. Thus, aprocessing device may identify the type of virtual content, for examplebased on the format of the virtual content, based on metadata associatedwith the virtual content, on resources required to process and/or renderthe virtual content (e.g., memory, CPU, and communications bandwidth),on latency experienced when rendering the virtual content, on timingrestrictions regarding the display of the virtual content, and any otheridentifiable characteristic of the virtual content. The type of virtualcontent (e.g., category, context, format, priority level) beingdisplayed in a given display region may be used to determine the firstand/or second duty cycle configurations and/or which duty cycleconfiguration to apply for displaying virtual content. For example, avirtual text document may be a different type of virtual content thatvirtual image, or video content. As another example, virtual contentassociated with an email application receiving text notifications inreal-time may be a different type (e.g., urgent text) than virtualcontent associated with a dormant graphic editing application displayinggraphics (e.g., non-urgent graphics). Thus, the urgent text may bedisplayed using a higher duty cycle configuration than the non-urgentgraphics. As yet another example, virtual content consumed during workhours (e.g., associated with a work context) may be a different typethan virtual content consumed after working hours (e.g., associated witha personal context). Thus, content associated with work may be displayedusing a higher duty cycle configuration that content associated withpersonal matters.

By way of a non-limiting example, turning to FIG. 6 , display region 604may include virtual widget 114C providing daily weather updates asgraphic content, and virtual widget 114D providing minute-by-minute textnotifications. Processing device 460 (FIG. 4 ) may identify thedifferent types of content displayed by virtual widgets 114C and 114D(e.g., graphic once per day versus text minute-by-minute) and maydetermine to display virtual widget 114D according to duty cycleconfiguration 610 (e.g., 60%), and virtual widget 114C according to dutycycle configuration 612 (e.g., 20%).

Some embodiments may further involve determining ambient illuminationconditions, and wherein at least one of the first duty cycleconfiguration and the second duty cycle configuration is determinedbased on the determined ambient illumination conditions. The term“ambient illumination conditions” may refer to the light that isavailable or present in an environment. An ambient illuminationcondition may involve one or more of the direction, intensity, color,quality, and/or the contrast-producing effect of light. For example, asource of light such as a window opening to daylight, a lamp, anelectronic display, or any other lighting appliance (e.g., turned on),as well as a physical object casting a shadow, the color of the walls,ceiling and floor, the presence of a mirror, and any other physicalobject affecting the available light may contribute to the ambientillumination conditions. The terms “determining”, and “duty cycleconfiguration” may be understood as defined earlier. The ambientillumination conditions may be determined, for example, by analyzing oneor more images captured by a camera (e.g., by a processor), by a lightmeter, an ambient light sensor (e.g., including one or morephototransistors, photodiodes, and photonic integrated circuits), or alux meter (e.g., configured with a mobile phone), or any other type ofambient light detector positioned in proximity to the extended realityenvironment. According to some embodiments, determining ambientillumination conditions may include determining the source of light(e.g., a window versus a LED lamp or screen), for example based on theluminance, the spectrum (e.g., detectable by a spectrophotometer).According to some embodiments, determining the ambient illuminationconditions may include determining properties of a light source, such asthe size, the direction, the presence of objects reflecting, absorbing,dispersing, and/or blocking the light source, and any other factoraffecting the light source.

Thus, the ambient illumination conditions in the extended realityenvironment (e.g., and the different display regions included therein)may be used to determine the first and/or second duty cycleconfigurations and/or which duty cycle configuration to apply fordisplaying virtual content. For example, a lower duty cycleconfiguration may be used to display virtual content in a shadowedregion of a room (e.g., because less contrast may be needed to discernthe virtual content), and a higher duty cycle configuration may be usedto display virtual content in a brightly lit region of the room (e.g.,because greater contrast may be needed to discern the virtual content).As another example, a higher duty cycle configuration may be applied todisplay virtual content during the day when the ambient illumination isprimarily due to sunlight, and a lower duty cycle configuration may beapplied to display virtual content at night when the ambientillumination is primarily due to artificial lighting. As yet anotherexample, while a curtain is drawn (e.g., open) allowing daylight topenetrate the physical space of the extended reality environment, ahigher duty cycle configuration may be used to display virtual content(e.g., to provide greater contrast to discern the virtual contentdisplayed in a well-lit area), and when the curtain is closed, a lowerduty cycle configuration may be used to display virtual content (e.g.,because less contrast may be needed to discern the virtual contentdisplayed in a darkened area).

By way of a non-limiting example, turning to FIG. 6 , camera 626positioned in extended reality environment may detect that displayregion 602 is situated in a well-lit area (e.g., exposed to daylight).In response, processing device 460 (FIG. 4 ) may determine to use dutycycle configuration 808 (e.g., 60%) to display virtual content indisplay region 602, e.g., to provide greater contrast for user 100 todiscern the virtual content. Conversely, camera 626 may detect thatdisplay region 604 is situated in a darkened area (e.g., due to a shadowcast by a physical object). In response, processing device 460 maydetermine to use duty cycle configuration 812 (e.g., 20%) to displayvirtual content in display region 604, e.g., because less contrast maybe needed.

Some embodiments may involve estimating a physical condition of a wearerof the wearable extended reality appliance, and wherein at least one ofthe first duty cycle configuration and the second duty cycleconfiguration is determined based on the estimated physical condition ofa wearer. The term “estimating” may include an approximation orassessment, e.g., based on analysis, calculations and/or inference ofmeasured data. The estimating may be facilitated by artificialintelligence, inference, statistical analysis, machine and/or deeplearning, extrapolation, clustering, and any other technique forperforming estimations. The term “physical condition” may refer to thephysiological state of the body or bodily functions of a user. Forexample, fatigue, nausea, eye strain, head, back and/or neck pain,posture, nervousness, agitation, illness, or any other physiologicalcondition affecting the physical condition of the user. The physicalcondition of the wearer may be estimated for example by processorreceiving data from a sensor configured to detect one or more biomarkers(e.g., heart or breathing rate, yawning, blinking frequency or eye openand close ratio (EOCR), the percentage of eyelid closure over the pupilover time (PERCLOS), blood pressure, oxygen level in exhaled air, or anyother biological indication of a physiological state) detected by one ormore sensors provided in the extended reality environment. For example,a smart watch worn by the user may detect heart and/or breathing rate. Acamera may capture images of the user yawning, head nodding head, or eyeclosing or rubbing and may provide the images to a processing device forimage analysis. An IMU configured with a pair of smart glasses maydetect a nodding motion of the wearer. The terms “wearer of the wearableextended reality appliance”, “determining”, and “duty cycleconfiguration” may be understood as defined earlier. Thus, the physicalor physiological state of the wearer of the wearable extended realityappliance may be used to determine the first and/or second duty cycleconfigurations and/or which duty cycle configuration to apply fordisplaying virtual content. For example, the duty cycle configurationmay be lowered if the wearer is determined to be agitated (e.g., basedon detecting distracted or jerky motions), fatigued, or suffering fromneck or back strain (e.g., by a camera capturing the wearer yawning,rubbing his eyes, or slouching). As another example, the duty cycleconfiguration may be increased if the wearer is determined to be alertand focused, e.g., based on an upright posture and a low PERCLOS level.

For example, during a learning period, a machine learning algorithm maydetect a pattern of behavior for a wearer and may receive feedback fromthe wearer allowing the machine learning algorithm to learn a scheduleof the wearer. The machine learning algorithm may use the schedule andfeedback to identify signs indicating the physical condition of thewearer, such as fatigue, stress, anxiety, nausea, a migraine, and anyother physical condition that may be alleviated or facilitated byadjusting the duty cycle. A processing device may use the identifiedsigns to modify the duty cycle configuration to accommodate thephysiological needs of the wearer. For example, if the wearer isdetermined to be suffering from fatigue (e.g., based on the detectedbreathing rate and PERCLOS level), the duty cycle may be reduced,similarly if the wearer is determined to be alert and energetic (e.g.,based on reaction time to displayed content), the duty cycle may beincreased.

By way of a non-limiting example with reference to FIG. 7 , camera 626may capture images of head nodding and eye closing by user 00,concurrently with motion sensor 473 sensing a nodding motion of the headof user 100. Processing device 460 (FIG. 4 ) may receive image data fromcamera 626 and the sensed motion data from sensor 473 and analyze theimage and motion data to determine that user 100 is experiencingdrowsiness. For example, processing device 460 may enlist a machinelearning engine to identify the nodding head motion and the closing ofthe eyes with sleepiness. In response, processing device 460 may modifythe duty cycle configuration used to display virtual content on virtualscreen 112 from duty cycle configuration 704 (e.g., 60%) to duty cycleconfiguration 708 (e.g., 20%).

Some embodiments may involve receiving an indication of a hardwarecondition of the wearable extended reality appliance, and wherein atleast one of the first duty cycle configuration and the second dutycycle configuration is determined based on the hardware condition of thewearable extended reality appliance. The terms “receiving”, “wearableextended reality appliance”, and “duty cycle configuration” may beunderstood as described earlier. The term “indication” may include asignal, sign, marker, measurement, or any other type of evidenceconveying a situation, state, or condition. The term “hardwarecondition” may include a state of a hardware component in the wearableextended reality appliance, such as the amount of available power in abattery, the processing load allocated to a processor, the availablememory or communications bandwidth, the temperature of an electroniccomponent, and any other measure of one or more hardware components ofthe wearable extended reality appliance. For example, a processingdevice may monitor available memory (e.g., stack, buffers, queues, RAM),communications bandwidth (e.g., for internal buses and externalcommunications channels), communication and processing latencies,temperature of electronic components, and any other hardware indication.The processing device may receive one or more indications of thehardware condition by polling various electronic components and/orreceiving one or more interrupt notifications, such as a buffer or stackoverflow notification, a NACK notification (e.g., a timeout afterexceeding a latency limit), an overheating warning from a thermometermonitoring the temperature of one or more electronic components, and/orby detecting a processing latency. The warnings may be issued based onpredefined thresholds for a given hardware configuration, e.g., based onrecommended specifications. Thus, the state of one or more hardwarecomponents included in the wearable extended reality appliance may beused to determine the first and/or second duty cycle configurationsand/or which duty cycle configuration to apply for displaying virtualcontent. For example, upon detecting a low battery level, or a highprocessing load allocation, the duty cycle configuration may be reduced,whereas upon detecting connection to a wall outlet and/or a lowprocessing load, the duty cycle configuration may be increased.

By way of a non-limiting example, a temperature sensor (e.g., othersensor 475 of FIG. 4 ) provided with wearable extended reality appliance110 may detect a temperature of processing device 460 and provide thetemperature reading to processing device 460. Processing device 460 maycompare the temperature reading to a predefined recommended temperaturelimit and may determine that processing device 460 is overheated, e.g.,due to a high processing load for displaying graphical virtual contentin virtual screen 112. In response, processing device 460 may reduce theduty cycle for displaying the virtual content to duty cycleconfiguration 612 (e.g., 20%) from duty cycle configuration 610 (e.g.,60%) to allow processing device 460 to cool to the recommendedtemperature limit.

Some embodiments may further involve identifying a virtual event, andwherein at least one of the first duty cycle configuration and thesecond duty cycle configuration is determined based on the identifiedvirtual event. The terms “identifying”, and “duty cycle configuration”may be understood as described earlier. The term “virtual event” mayinclude an occurrence of an action, activity, or any other change ofstate that is implemented via a computer-generated medium and may notexist outside the computer-generated medium (e.g., in the real, physicalworld detached from a computer). A virtual event may be identified, forexample, based on the processing load of a processing unit (e.g., aGPU), the status of memory resources (e.g., buffers, queues, and stacks,RAM), retrieval of data from a particular location in memory, receivingof data from a specific external source, as a notification from anexternal device or an event listener (e.g., configured with an operatingsystem of the wearable extended reality appliance), as latencyexperienced in processing threads other than the virtual event, aresponse of the wearer of the extended reality appliance, and any otherindication of a virtual event. A processing device may identify thevirtual event, for example by polling one or more memory resources,monitoring the status of internal buses and/or external communicationschannels (e.g., by checking latency and time-outs), receiving aninterrupt event from an event listener, and any other method foridentifying the virtual event. Additionally, or alternatively, a virtualevent may be identified based on feedback from a user of an extendedreality appliance, such as a head motion, voice command and/or action byan electronic pointing device in response to or related to the virtualevent. Thus, one or more synthesized (e.g., virtual) events in theextended reality environment may be used to determine the first and/orsecond duty cycle configurations and/or which duty cycle configurationto apply for displaying virtual content. For example, the entry of avirtual avatar entering the room may cause virtual content other thanthe avatar to be displayed according to a lower duty cycle configurationand the avatar to be displayed according to a higher duty cycleconfiguration. As another example, the sharing (e.g., sending orreceiving) of content may cause memory buffer overflow and/or anoverload on a bus system and may trigger a change in the duty cycleconfiguration, e.g., to a lower duty cycle to alleviate processing load.

By way of a non-limiting example, turning to FIG. 6 , user 100 mayreceive an electronic notification (e.g., the occurrence of a virtualevent) associated with virtual widget 114D. In response, processingdevice 460 (FIG. 4 ) may increase the duty cycle for displaying widget114D from duty cycle configuration 612 (e.g., 20%) to duty cycleconfiguration 610 (e.g., 60%).

According to some embodiments, the first duty cycle configuration forthe first display region and the second duty cycle configuration for thesecond display region are determined for a first time period, and theoperations further include determining at least one updated duty cycleconfiguration for the first display region and the second display regionfor a second time period following the first time period. The terms“duty cycle configuration”, “display region”, and “determining” may beunderstood as described earlier. The term “time period” may refer to aduration, length of time for an activity, condition or state (e.g.,measured in seconds, minutes, hours, and/or days), a particular time ofday (e.g., morning, afternoon, evening or night), a particular day ordays of the week (e.g., weekdays versus weekends or holidays), or anyother measure of time. The term “updated” may refer to amended, renewedor revised. Thus, a time-based criterion may be used to determine and/orupdate the first and/or second duty cycle configurations and/or whichduty cycle configuration to apply for displaying virtual content. Forexample, virtual content may be displayed according to a lower dutycycle configuration during morning hours when the wearer of the wearableextended reality appliance is alert. In the afternoon (e.g., followingthe morning) when the user is fatigued, the duty cycle may be updated toa higher duty cycle configuration, e.g., to draw the wearer's focus. Inthe evening (e.g., following the afternoon), the duty cycle may beupdated yet again to a lower duty cycle configuration, e.g., to allowthe wearer to relax during a shutdown ritual.

By way of a non-limiting example, turning to FIG. 6 , during daylighthours when the ambient lighting is due to natural sunlight, processingdevice 460 (FIG. 4 ) may display virtual content on virtual screen 112according to duty cycle configuration 610 (e.g., 60%), for example toprovide a more intense display to overcome intense daylightillumination. During the evening hours, e.g., when the ambient lightingis based on an artificial light source, such as a light bulb that isdimmer than sunlight, processing device 460 may display virtual contenton virtual screen 112 according to duty cycle configuration 612 (e.g.,20%), for example to conserve energy because a less intense display maybe sufficient.

According to some embodiments, the operations further includedetermining when to end the first time period based on detection of anevent. The term “determining” and “time period” may be understood asdescribed above. The term “when to end the first time period” may beunderstood as a boundary for the time period, for example a point intime when the first time period terminates, and a new time periodcommences. For example, determining when to end the first time periodmay include calculating, identifying, specifying, setting, or assigninga boundary of for the first time period. For example, a processingdevice (e.g., configured with the wearable extendible reality appliance)may be configured to calculate, specify, set, or assign a point in timefor the termination of the first time period based on detection of anevent. The term “event” may refer to an occurrence of an action,activity, change of state, or any other type of development or stimulus,for example detectable by a processing device. The source of the eventmay be internal or external to the wearable extendible realityappliance.

For example, an internal event may include a signal relating to a stateof a component of the wearable extendible reality appliance (e.g.,temperature, available communication and/or processing bandwidth,available battery power or memory, or any other criteria relating to theoperation of the wearable extendible reality appliance). Internal eventsthat may trigger a processing device to terminate the first time periodmay include, for example, the internal temperature of the processingdevice exceeding a predefined limit, the power remaining in a batteryfor the processing device falling below a predefined threshold, or amemory buffer overflowing.

Examples of an external event may include an alert, trigger or signalreceived from an external computing device or peripheral device (e.g.,configured with a sensor such as an optical, IR, acoustic, vibration,temperature, heat, humidity, electric and/or magnetic, or any other typeof sensor), a user (e.g., as a user input) or any other type of externalstimulus. The user input may include input via an input device (e.g.,keyboard, electronic pointing device, touch-based device), a voicecommand, a gesture (e.g., eye via an eye tracker, head, hand, body), orany other type of user input. For example, external events that maytrigger a processing device to terminate the first time period mayinclude receiving a notification of a scheduled calendar event,receiving a timeout notification (e.g., NACK) from an external device,completing the receiving of data from an external source, or receivingan external warning to update system software, or install protectivemeasures against malware.

Thus, the termination of the first time period may be based on detectingan internal and/or external event. For example, a timer issuing an alertat a predefined hour, or a microphone sensing a child returning homefrom school may be used to determine the termination of the first timeperiod. As another example, an application invoked by the wearer, or thereceiving of a notification from another computing device (e.g., anemail or electronic message) detected by an event listener may be usedto determine the termination of the first time period.

By way of a non-limiting example, turning to FIG. 6 , processing device460 (FIG. 4 ) may apply duty cycle configuration 610 to display virtualcontent on virtual screen during a first time period. Upon user 100receiving a notification (e.g., associated with virtual widget 114D)relating to an urgently scheduled meeting, processing device 460 (FIG. 4) may analyze the notification and determine to terminate the first timeperiod and initiate the second time period. Processing device 460 mayswitch the duty cycle for displaying virtual content on virtual screen112 to correspond to duty cycle configuration 612, e.g., for the secondtime period, for example to allow user 100 to prepare for the meeting.

According to some embodiments, the at least one updated duty cycleconfiguration includes a single duty cycle configuration for both thefirst display region and the second display region. The term “updated”,“duty cycle configuration”, and “display region” may be understood asdescribed earlier. The term “single” may refer to sole or only. Thus,after the first time period terminates, only one (e.g., single) dutycycle configuration may be used to display content in the first andsecond display regions, e.g., during the second time period followingthe first time period. For example, after a predetermine hour (e.g.,midnight) any content displayed via the wearable extended realityappliance (e.g., in the frst and second display regions) may bedisplayed according to the same duty cycle configuration, such as alower duty cycle configuration to conserve energy.

By way of a non-limiting example, turning to FIG. 6 , after a timeperiod (e.g., a predetermined time period) of operation, processingdevice 460 (FIG. 4 ) may determine that power source 440 providing powerto operate wearable extended reality appliance 110 is low on power, forexample power source 440 may be a battery and wearable extended realityappliance 110 may be a wireless appliance. In response, processingdevice 460 may update the duty cycle configuration to a lower duty cycle(e.g., 20%) for virtual content displayed in any of display regions 602and 604, e.g., to conserve power.

According to some embodiments, the at least one updated duty cycleconfiguration includes a first updated duty cycle configuration for thefirst display region and a second updated duty cycle configuration forthe second display region. The terms “updated duty cycle configuration”,and “display region” may be understood as described earlier. Thus, afterthe first time period terminates, each display region may be associatedwith a different updated duty cycle configuration. The duty cycleconfigurations may be increased or decreased by the same or differentamounts. For example, both duty cycle configurations may be increased(e.g., both increased by 10%), one duty cycle configuration may beincreased (e.g., by 5%) and the other duty cycle configuration may bedecreased (e.g., by 20%), or both duty cycle configurations may bedecreased (e.g., one by 5% and the other by 15%). For example, duringthe first time period, content may be displayed in the first displayregion according to an 80% duty cycle configuration, and content may bedisplayed in the second display region according to a 60% duty cycleconfiguration. When the first time period terminates, the duty cycleconfiguration may be updated for both the first and second displayregions, e.g., by reducing the duty cycle configuration for the firstdisplay region by 10% and by increasing the duty cycle configuration forthe second display region by 20%. Thus, during the second time period(e.g., following the first time period), content may be displayed in thefirst display region according to a 70% duty cycle configuration (e.g.,the first updated duty cycle configuration), and content may bedisplayed in the second display region according to an 80% duty cycleconfiguration (e.g., the second updated duty cycle configuration).

By way of a non-limiting example, turning to FIG. 6 , during the firsttime period, processing device 460 (FIG. 4 ) may display content indisplay region 602 according to duty cycle configuration 610 (e.g., 60%)and content in display region 604 according to duty cycle configuration612 (e.g., 20%). After the lapse of a predetermined time period (e.g.,the first time period), processing device 460 may determine that powersource 440 (e.g., a battery) is running low, and may update the dutycycle configurations for displaying content in each of display regions602 and 604, e.g., by reducing the duty cycle for each duty cycleconfiguration by half. Thus, during the second time period (e.g.,following the first time period), processor may display content indisplay region 602 according to a duty cycle configuration of 30% (e.g.,the first updated duty cycle configuration) and content in displayregion 604 according to a duty cycle configuration of 10% (e.g., thesecond updated duty cycle configuration).

According to some embodiments, the at least one updated duty cycleconfiguration includes a first updated duty cycle configuration for thefirst display region and a first portion of the second display regionand a second updated duty cycle configuration for a second portion ofthe second display region, the first portion of the second displayregion differs from the second region of the second display region. Theterms “updated duty cycle configuration”, “display region”, and“differs” may be understood as described earlier. Thus, the extendedreality environment may be divided into different display regions forthe first and second time periods such that the first and second displayregions include different sections of the extended reality environmentduring the first and second time periods. In other words, in addition toupdating the duty cycle configurations, the regions of the extendedreality environment included in each of the first and second displayregions may be updated.

For example, during the first time period, the first display region maybe limited to a virtual screen directly facing the user, and the seconddisplay region may include a section of the extended reality environmentadjacent to the virtual screen as well as a desktop, e.g., supporting akeyboard. During the first time period, content may be displayed in thefirst display region (including just the virtual screen) according tothe first duty cycle configuration, and in the second display region(including the area adjacent to the virtual screen and the desktop)according to the second duty cycle configuration. During the second timeperiod, the display regions may be divided up differently. For example,the first updated display region may now include the virtual screen andadditionally the desktop, and the second updated display region may nowinclude only the section adjacent to the virtual screen (e.g., withoutthe desktop). Content in the first updated display region (including thevirtual screen and desktop) may be displayed according to the firstupdated duty cycle configuration, and content in the second updateddisplay region (including only the section adjacent to the virtualscreen) may be displayed according to the second updated duty cycleconfiguration.

By way of a non-limiting example, turning to FIG. 6 , during the firsttime period, processing device 460 (FIG. 4 ) may display content indisplay region 602 (e.g., virtual screen 112) according to duty cycleconfiguration 610 (e.g., 60%) and content in display region 604 (e.g.,virtual widgets 114C and 114D) according to duty cycle configuration 612(e.g., 20%). When the first time period lapses, (e.g., in response to anotification associated with virtual widget 114D), processing device 460may update the first and second duty cycle configurations, for exampleby lowering the duty cycle for each by 10%. Thus, the first updated dutycycle configuration may now be 50% and the second updated duty cycleconfiguration may now be 10%. However, processing device 460 maydetermine that virtual widget 114D should be displayed according to thehigher of the two duty cycle configurations (e.g., 50%), e.g., to drawthe attention of user 100 to incoming notifications. Thus, processingdevice 460 may use the 50% duty cycle configuration (e.g., first updatedduty cycle configuration) to display virtual widget 114D (e.g., a firstportion of display region 604) and virtual screen 112 and may use the10% duty cycle configuration (e.g., second updated duty cycleconfiguration) to display virtual widget 114C (e.g., second portion ofdisplay region 604).

Some embodiments may provide a non-transitory computer readable mediumcontaining instructions that when executed by at least one processorcause the at least one processor to perform duty cycle controloperations for wearable extended reality appliances, the operations maycomprise: receiving data representing virtual content in an extendedreality environment associated with a wearable extended realityappliance; causing the wearable extended reality appliance to displaythe virtual content in accordance with a first duty cycle configuration;after causing the wearable extended reality appliance to display thevirtual content in accordance with the first duty cycle configuration,determining a second duty cycle configuration; and causing the wearableextended reality appliance to display the virtual content in accordancewith the second duty cycle configuration. The percent during which eachcycle of the display signal is set to “active” may be non-zero in boththe first duty cycle configuration and the second duty cycleconfiguration. The second duty cycle configuration may differ from thefirst duty cycle configuration. According to some embodiments, theoperations may further include determining when to switch from thedisplay in accordance with the first duty cycle configuration to thedisplay in accordance with the second duty cycle configuration based ondetection of an event, for example as described above.

According to some embodiments, the first duty cycle configuration forthe first display region includes a selection of different duty cyclesfor a display device associated with a left eye of a wearer of thewearable extended reality appliance and for a display device associatedwith a right eye of the wearer of the wearable extended realityappliance. The terms “duty cycle configuration”, “display region”, and“different” may be understood as described earlier. The term “selection”may refer to election or choosing an option from several options. Theterm “display device associated with a left eye” (e.g., or right eye)may refer to a device configured to accommodate vision correctiverequirements (e.g., to correct for one or more of emmetropia, myopia,hyperopia, and astigmatism) for the left or right eye, respectively. Forexample, the display device (e.g., for the left and/or right eye) mayinclude one or more optically lenses to adjust a view seen through thewearable extended reality appliance, for example to adjust the focus oflight onto the retina of the left and/or right eye and/or magnify animage. As another example, the display device (e.g., for the left and/orright eye) may include a coating or filter, such as an anti-reflectiveor polarized coating to reduce glare. As another example, the displaydevice (e.g., for the left and/or right eye) may include one or morephotosensitive materials (e.g., photochromic dyes) to block incomingultraviolet light.

For example, the wearer of the wearable extended reality appliance mayhave different vision corrective requirements for each eye (e.g., theleft eye may require correction for astigmatism and high myopia, and theright eye may require correction only for low myopia). Additionally, oralternatively, the wearer may wish to use the left eye to view contentup close and the right eye to see content from a distance. As anotherexample, the wearer of the wearable extendible reality appliance mayhave undergone cataract surgery in one eye. The wearable extendedreality appliance may thus include a different display device for eacheye, each display device accommodating the vision requirements of eacheye, e.g., to adjust the focus of light onto the retina of each eye,reduce glare, and/or filter certain wavelengths (e.g., ultravioletlight). When determining the duty cycle configuration, a different dutycycle configuration may be selected for each of the display devices,e.g., to accommodate the seeing requirements of each eye of the user.For example, a higher duty cycle configuration may be selected for thedisplay device associated with the higher myopia eye than the lowermyopia eye.

By way of a non-limiting example with reference to FIG. 6 , the left eyeof user 100 may have undergone surgery to remove a cataract and correctfor vision impairment, whereas the right eye may have a myopia of −4diopters. Wearable extended reality apparatus 110 may be a pair of smartglasses allowing user 100 to view the physical environmentsimultaneously with virtual content. The smart glasses may include asmart left lens (e.g., display device associated with the left eye) thatis clear (e.g., no vision correction) with an anti-LV coating, and asmart right lens (e.g., display device associated with the right eye)correcting for the myopia but without any anti-UV coating. Whendetermining the duty cycle configuration, processing device 460 (FIG. 4) may select a different duty cycle configuration for the smart leftlens and the smart right lens to accommodate the different correctiveneeds for each eye. For example, a lower duty cycle configuration may beused for the left eye to ease eye strain following cataract surgery, anda higher duty cycle configuration for the right eye to provide a brightdisplay.

Some embodiments may provide a system for duty cycle control forwearable extended reality appliances, the system including at least oneprocessor programmed to: receive data representing virtual content in anextended reality environment associated with a wearable extended realityappliance; identify in the extended reality environment a first displayregion and a second display region separated from the first displayregion; determine a first duty cycle configuration for the first displayregion; determine a second duty cycle configuration for the seconddisplay region, wherein the second duty cycle configuration differs fromthe first duty cycle configuration; and cause the wearable extendedreality appliance to display the virtual content in accordance with thedetermined first duty cycle configuration for the first display regionand the determined second duty cycle configuration for the seconddisplay region.

For example, turning to FIG. 6 in conjunction with FIG. 4 , system 600may include processing device 460, which may be programmed to receivedata representing virtual content in extended reality environment 620associated with wearable extended reality appliance 110. Processingdevice 460 may identify in extended reality environment 620 a firstdisplay region 602 and a second display region 604, separated from firstdisplay region 602. Processing device 460 may be determined duty cycleconfiguration 610 (e.g., 60%) for display region 602 and duty cycleconfiguration 612 for display region 604, where duty cycle configuration612 differs from duty cycle configuration 610. Processing device 460 maycause wearable extended reality appliance 110 to display the virtualcontent in accordance with duty cycle configuration 610 for displayregion 602 and duty cycle configuration 612 for the display region 604.

FIG. 9 illustrates a block diagram of an example process 900 forcontrolling a duty cycle for wearable extended reality appliancesconsistent with embodiments of the present disclosure. In someembodiments, process 900 may be performed by at least one processor(e.g., processing device 460 of extended reality unit 204, shown in FIG.4 ) to perform operations or functions described herein. In someembodiments, some aspects of process 900 may be implemented as software(e.g., program codes or instructions) that are stored in a memory (e.g.,memory device 411 of extended reality unit 204, shown in FIG. 4 ) or anon-transitory computer readable medium. In some embodiments, someaspects of process 900 may be implemented as hardware (e.g., aspecific-purpose circuit). In some embodiments, process 900 may beimplemented as a combination of software and hardware.

Referring to FIG. 9 , process 900 may include a step 902 of receivingdata representing virtual content in an extended reality environmentassociated with a wearable extended reality appliance. As describedearlier, data formatted for displaying virtual content in an extendedreality environment via a wearable extended reality appliance may bereceived. For example, the data may be generated by a processorconfigured with the wearable extended reality appliance or may bereceived from an extemal computing device via a transceiver.

By way of a non-limiting example with reference to FIG. 6 , processingdevice 460 (FIG. 4 ) of wearable extended reality appliance 110 mayreceive data representing virtual content (e.g., virtual widgets 114Dand 114D, and virtual screen) in extended reality environment 620associated with a wearable extended reality appliance 110.

Process 900 may include a step 904 of identifying in the extendedreality environment a first display region and a second display regionseparated from the first display region may be identified. As describedearlier, the extended reality environment may include a virtual displaygenerated, for example, by a wearable extended reality appliance. Aprocessing device may be configured to identify one or more regions inthe virtual display. For example, a processing device may identify afirst region associated with work-related content, and a second regionassociated with personal content.

By way of a non-limiting example, processing device 460 (FIG. 4 ) ofwearable extended reality appliance 110 may identify in extended realityenvironment 620 (FIG. 6 ), display regions 602 and display region 604separated from first display region 602. Display region 602 may beassociated with displaying work-related documents, such as charts andtext documents for editing. Display region 604 may be associated withdisplaying virtual accessories to assist user 100, such as virtualwidgets 114C and 114D providing weather updates and notifications,respectively.

Process 900 may include a step 906 of determining a first duty cycleconfiguration for the first display region. As described earlier, aprocessing device may be configured to determine a duty cycleconfiguration for the first display region, for example to adjust theintensity of the display, and/or manage power consumption. For example,if the first display region is for work-related content, a relativelyhigh duty cycle configuration (e.g., 80%) may be determined, e.g., todraw the attention of the user. By way of a non-limiting example,processing device 460 (FIG. 4 ) of wearable extended reality appliance110 may determine duty cycle configuration 610 for display region 602 inFIG. 6 . For example, processing device 460 may determine that displayregion 602 is currently the primary area of focus for user 100 and mayapply a duty cycle configuration of 60%.

Process 900 may include a step 908 of determining a second duty cycleconfiguration for the second display region, where the second duty cycleconfiguration differs from the first duty cycle configuration. Asdescribed earlier, the virtual display generated, for example, by awearable extended reality appliance may include first and second displayregions. A processing device may be configured to determine a differentduty cycle configuration for each display region, for example toseparately adjust the intensity of the display and/or power consumptionfor each display region. For example, the second display region may bedesignated for personal content and the first display region may bedesignated for work related content. While the wearer of the extendedreality applicant is engaged in work, the processing device maydetermine a lower duty cycle configuration (e.g., 40%) for the seconddisplay region, e.g., to facilitate the wearer in maintaining focus onthe work-related content.

By way of a non-limiting example, processing device 460 (FIG. 4 ) ofwearable extended reality appliance 110 may determine duty cycleconfiguration 612 (e.g., 20%) for display region 604, which differs fromduty cycle configuration 610 (e.g., 60%) determined for display region602 of FIG. 6 . The different duty cycle configurations 610 and 612applied to each display region 602 and 604, respectively, may facilitateuser 100 in concentrating on display region 602, and avoid beingdistracted by updates from virtual widgets 114C and 114D in displayregion 604.

Process 900 may include a step 910 of causing the wearable extendedreality appliance to display the virtual content in accordance with thedetermined first duty cycle configuration for the first display regionand the determined second duty cycle configuration for the seconddisplay region. As described earlier, a processing device may beconfigured to control the display of virtual content in differentdisplay regions of a virtual display generated by a wearable extendedreality appliance. For example, the processing device may control thedisplay of the virtual content by determining the duty cycleconfiguration for applying to each display region. Additionally, theprocessing device may cause the wearable extended reality device todisplay virtual content in each display region according to eachdetermined duty cycle configuration. For example, the processing devicemay control signals (e.g., by controlling the level, intensity,frequency, timing, power level, phase, and any other signal attributeaffecting the duty cycle) carried to each display region via one or moredata and/or power lines coupling the processing device to each displayregion of the display of the wearable extended reality appliance.Consequently, content in the first display region may be displayedaccording to the first duty cycle configuration, and content in thesecond display region may be displayed according to the second dutycycle configuration. Returning to the example above, work-relatedcontent may be displayed in the first display region according to an 80%duty cycle and personal content may be displayed in the second displayregion according to a 40% duty cycle.

By way of a non-limiting example, processing device 460 (FIG. 4 ) ofwearable extended reality appliance 110 may cause wearable extendedreality appliance 110 to display virtual screen 112 in display region604 of FIG. 6 according to duty cycle configuration 610 and virtualwidgets 114C and 114D in display region 606 according to duty cycleconfiguration 610.

Extended reality environments may include virtual and physical displayareas, such as virtual displays (e.g., bounded regions defining virtualscreens), and physical objects such as walls and surfaces. An extendedreality appliance may present virtual objects anywhere in the extendedreality environment, at differing distances from a user. Users may wishto organize virtual objects, such as to unclutter a virtual display areaor to change a presentation mode for content. For example, contentextracted from a virtual display may be modified (e.g., magnified) whenpresented outside the virtual display.

The description that follows includes references to smart glasses as anexemplary implementation of a wearable extended reality appliance. It isto be understood that these examples are merely intended to assist ingaining a conceptual understanding of disclosed embodiments, and do notlimit the disclosure to any particular implementation for a wearableextended reality appliance. The disclosure is thus understood to relateto any implementation for a wearable extended reality appliance,including implementations different than smart glasses.

Some embodiments involve a non-transitory computer readable mediumcontaining instructions that when executed by at least one processorcause the at least one processor to perform operations for extractingcontent from a virtual display. The term “non-transitorycomputer-readable medium” may be understood as described earlier. Theterm “instructions” may refer to program code instructions that may beexecuted by a computer processor. The instructions may be written in anytype of computer programming language, such as an interpretive language(e.g., scripting languages such as HTML and JavaScript), a procedural orfunctional language (e.g., C or Pascal that may be compiled forconverting to executable code), object-oriented programming language(e.g., Java or Python), logical programming language (e.g., Prolog orAnswer Set Programming), or any other programming language. In someembodiments, the instructions may implement methods associated withmachine learning, deep learning, artificial intelligence, digital imageprocessing, and any other computer processing technique. The term“processor” may be understood as described earlier. For example, the atleast one processor may be one or more of server 210 of FIG. 2 , mobilecommunications device 206, processing device 360 of FIG. 3 , processingdevice 460 of FIG. 4 , processing device 560 of FIG. 5 ), and theinstructions may be stored at any of memory devices 212, 311, 411, or511, or a memory of mobile device 206. The term “content” may refer todata or media formatted for presenting information to a user via, forexample, an interface of an electronic device. Content may include, forexample, any combination of data formatted as alphanumerical text, imagedata, audio data, video data, and any other data type for conveyinginformation to a user. The term “extracting content” may refer tocontent that is separated or pulled out, e.g., from other content. Theterm “virtual display” may refer to a virtual object mimicking and/orextending the functionality of a physical display screen, as describedearlier. A virtual display may function as a container (e.g., 2D frameor 3D box) for multiple other virtual objects.

For example, at least one processor may display virtual content,including multiple virtual objects, via a wearable extended realityappliance. One of the virtual objects may be a virtual displaycontaining one more of the other virtual objects, e.g., as a frame orbox encasing a group of the other virtual objects. The at least oneprocessor may execute instructions to separate or pull out (e.g.,extract) one or more virtual objects (e.g., content) of the group ofobjects contained in the virtual display. As an example, a virtualdisplay may contain a group of virtual objects, including a virtualdocument and several virtual widgets and the at least one processor mayremove one of the widgets (e.g., extract content) from the virtualdisplay.

By way of a non-limiting example, FIG. 10 illustrates an exemplaryenvironment depicting a user 1016 of a wearable extended realityappliance (e.g., a pair of smart glasses 1006) moving content between avirtual display 1002 and an extended reality environment 1004. Extendedreality environment 1004 may be generated via a system (e.g., system 200of FIG. 2 ). Virtual display 1002 may serve as a frame containing agroup 1050 of multiple virtual objects, such as a virtual document 1008,virtual widgets 1010 inside a virtual menu bar 1024, a virtual workspace1012, and a virtual house plant 1014. A user 1016 donning smart glasses1006 may interface with content displayed by smart glasses 1006, e.g.,via gestures, voice commands, keystrokes on a keyboard 1018, a pointingdevice such as an electronic mouse 1022, or any other user interfacingmeans. Processing device 460 (FIG. 4 ) may extract content from virtualdisplay 1002, (e.g., in response to input from user 1016). As anexample, processing device 460 may extract virtual house plant 1014 fromvirtual display 1002.

Some embodiments involve generating a virtual display via a wearableextended reality appliance, wherein the virtual display presents a groupof virtual objects and is located at a first virtual distance from thewearable extended reality appliance. The terms “virtual display” and“wearable extended reality appliance” may be understood as describedearlier. The term “generating” may refer to producing, synthesizing,constructing, or creating. The term “virtual object” may refer to avisual rendition of an item by a computer. Such an object may have anyform, such as an inanimate virtual object (e.g., icon, widget, document;representation of furniture, a vehicle, real property, or personalproperty; an animate virtual object (e.g., human, animal, robot); or anyother computer-generated or computer supplied representation) asdescribed earlier. For example, an extended reality appliance mayproduce (e.g., generate) a virtual object by activating selected pixelsto render the virtual object overlaid against the physical environmentsurrounding the user and viewable through transparent portions of theviewer. The term “presents” may refer to displaying, demonstrating, orcommunicating, e.g., to convey information encoded as text, image data,audio data, video data, haptic data, or any other communications medium.For example, an electronic display may present information visually, anda speaker may present information audibly. The term “group of virtualobjects” may refer to a collection or cluster of one or more virtualobjects. As an example, one or more virtual objects may be groupedinside a virtual display of the wearable extended reality appliance. Theterm “located” may refer to a station, placement, or position of anobject. The term “virtual distance” may refer to a spatial separation orgap between a wearable extended reality appliance and one or morevirtual objects or between the one or more virtual objects, as perceivedby a user wearing the wearable extended reality appliance. The distancemay be along a two-dimensional plane (e.g., the floor), or through athree-dimensional volume (e.g., accounting for the height of thesurrounding physical environment in addition to floor distance). Thedistance may be absolute (e.g., relative to the Earth, or based on GPScoordinates), or relative (e.g., with respect to an object in theextended reality environment). The distance may be relative to aphysical object, a virtual object, the wearable extended realityappliance, and/or the user (e.g., the distance may be relative to morethan one reference). As an example, the wearable extended realityappliance may present multiple virtual objects inside a virtual displayappearing as though located at particular spatial separations from theuser (e.g., at arm's length, or on a wall opposite the user).

By way of a non-limiting example, in FIG. 10 , smart glasses 1006 mayproduce (e.g., generate) virtual display 1002 as a frame containinggroup 1050 of multiple virtual objects, such as virtual document 100,virtual widgets 1010 inside virtual menu bar 1024, virtual workspace1012, and virtual house plant 1014. Smart glasses 1006 may displayvirtual display 1002 to appear at a virtual distance D1 from user 1016.For example, D1 may be measured relative to a 3D coordinate system 1028as the distance from smart glasses 1006 to the bottom left corner ofvirtual display 1002.

Some embodiments involve generating an extended reality environment viathe wearable extended reality appliance, wherein the extended realityenvironment includes at least one additional virtual object presented ata second virtual distance from the wearable extended reality appliance.The term “extended reality environment,” e.g., also referred to as“extended reality,” “extended reality space,” or “extended environment,”may refer to all types of real-and-virtual combined environments andhuman-machine interactions at least partially generated by computertechnology, as described earlier. For example, an extended realityenvironment may encompass the field-of-view of a user donning a wearableextended reality appliance and may include the physical environmentsurrounding the user as well as virtual content superimposed thereon. Aprocessing device of the wearable extended reality appliance may produceor generate an extended reality environment by selectively activatingcertain pixels of a viewer of the wearable extended reality appliance torender virtual content overlaid on the physical environment viewable viatransparent portions of the viewer.

For example, the extended reality environment created by the wearableextended reality appliance may contain multiple virtual objects. Somevirtual objects may be grouped inside a virtual display positioned at afirst perceived (e.g., virtual) distance from the user. The wearable maydisplay at least one additional virtual object at a second virtualdistance from the user. The first and second virtual distances may bemeasured across a 2D plane (e.g., the floor), or through a 3D space ofthe extended reality environment (e.g., to account for a height ofdisplayed content). The first and second virtual distances may be thesame or different (e.g., larger, or smaller). As an example, the firstand second virtual distances may be substantially the same as measuredacross the floor (e.g., in 2D) but may differ along the heightdimension. As another example, the first and second virtual distancesmay differ in one or more directions as measured across the floor (e.g.,in 2D) and also along the height dimension.

As an example, the at least one processor may determine the first and/orsecond virtual distances based on a 3D spatial map of the physicalenvironment surrounding the wearable extended reality appliance (e.g.,as a mesh of triangles or a fused point cloud). The first and/or secondvirtual distances may be determined based on one or more physicalobjects in the extended reality environment, data stored in memory(e.g., for the location of stationary objects), predicted behaviorand/or preferences of the wearer of the wearable extended realityappliance, ambient conditions (e.g., light, sound, dust), and any othercriterion for determining a distance for presenting virtual objects. Forexample, a physical object may be detected via sensors interface 472 ofFIG. 4 ).

By way of a non-limiting example, in FIG. 10 , smart glasses 1006 maygenerate extended reality environment 1004 to include virtual content,such as virtual objects grouped inside virtual display 1002 and virtualmobile phone 1026. Processing device 460 (FIG. 4 ) may display thevirtual content overlaid on the surrounding physical environment seen byuser 1016 through smart glasses 1006. Processing device 460 may displayvirtual display 1002 to appear at a distance D1 from user 1016 and maydisplay virtual mobile phone 1026 to appear at a distance D2 from user1016.

Some embodiments involve receiving input for causing a specific virtualobject from the group of virtual objects to move from the virtualdisplay to the extended reality environment. The term “receiving” mayrefer to accepting delivery of, acquiring, retrieving, obtaining, orotherwise gaining access. The term “input” may include information, suchas a stimulus, response, command, or instruction, e.g., targeted to aprocessing device. For example, an input provided by a user may bereceived by the at least one processor via an input interface (e.g.,input interface 430 of FIG. 4 and/or input interface 330 of FIG. 3 ), bya sensor associated with the wearable extended reality appliance (e.g.,sensor interface 470 or 370), by a different computing devicecommunicatively coupled to the wearable extended reality appliance(e.g., mobile device 206 and/or remote processing unit 208 of FIG. 2 ),or any other source of input. A user input may be provided via akeyboard, a touch sensitive screen, an electronic pointing device, amicrophone (e.g., as audio input or voice commands), a camera (e.g., asgesture input), or any other user interfacing means. An environmentalinput (e.g., relating to ambient noise, light, dust, physical objects,or persons in the extended reality environment) may be provided via oneor more sensors (e.g., sensor interface 470 or 370). A device input(e.g., relating to processing, memory, and/or communications bandwidth)may be received by a processing device (e.g., any of server 210 of FIG.2 , mobile communications device 206, processing device 360, processingdevice 460, or processing device 560 of FIG. 5 ).

The term “causing” may refer to invoking or triggering an action oreffect. For example, a user input to open an application may lead (e.g.,cause) at least one processor to open the application. As anotherexample, an ambient light level may be received (e.g., as anenvironmental input) leading to (e.g., causing) at least one processorto adjust the brightness of displayed content. The term “specificvirtual object from the group of virtual objects” may refer to adistinct, or particular virtual object out of a collection of multiplevirtual objects. The term “move” may refer to relocating or changing aposition. For example, a specific widget (e.g., a specific virtualobject) included in a group of virtual objects displayed inside avirtual display may be relocated (e.g., moved) to a different locationin the extended reality environment, external to the virtual display,e.g., in response to an input. The input may be a user input, e.g.,requesting to move the specific widget, an environmental input, e.g., anambient light setting affecting the visibility of displayed objects, adevice input, e.g., relating to the operation of the wearable extendedreality appliance, or any other type of input.

As an example, a wearable extended reality appliance may present avirtual display presenting a group of multiple virtual objects. An inputmay be received to cause a particular one of the multiple virtualobjects to be relocated to a different display location, outside thevirtual display. The input may be received from a user who may wish toview the specific virtual object from a distance nearer than the virtualdisplay. As another example, the input may be received from a sensordetecting an obstruction (e.g., bright light, or obstructing object)blocking the specific virtual object. As yet another example, the inputmay be received from a software application monitoring the density ofcontent displayed inside the virtual display.

Some embodiments include receiving the input from an image sensorindicative of a gesture initiated by a user of the wearable extendedreality appliance. The term “image sensor” may include a detector (e.g.,a camera) configured to capture visual information by converting lightto image data, as described earlier. The term “gesture” may refer to amovement or sequence of movements of part of the body, such as a hand,arm, head, foot, or leg to express an idea or meaning. A gesture may bea form of non-verbal or non-vocal communication in which visible bodilyactions or movements communicate particular messages. A gesture may beused to communicate in place of, or in conjunction with vocalcommunication. For example, raising a hand with the palm forward may bea hand gesture indicating to stop or halt an activity, and raising athumb with the fist closed may indicate approval. A gesture may bedetected as an input using an image sensor (e.g., image sensor 472 ofFIG. 4 ) and/or a motion detector (e.g., motion sensor 473) associatedwith the wearable extended reality appliance. In some examples, theinput from the image sensor (such as images and/or videos captured usingthe image sensor) may be analyzed (for example, using a gesturerecognition algorithm) to identify the gesture initiated by the user. Inone example, the gesture may be indicative of a desire of the user tocause the specific virtual object to move from the virtual display tothe extended reality environment. In one example, the gesture may beindicative of the specific virtual object and/or of a desired positionin the extended reality environment for the specific virtual object.

By way of a non-limiting example, reference is made to FIG. 11 , whichis substantially similar to FIG. 10 with a noted difference in FIG. 10 ,virtual house plant 1014 is presented inside virtual display 1002, andin FIG. 11 , version 1014A of virtual house plant 1014 is displayedexternal to virtual display 1002, as though resting on a desk top 1020.In FIG. 10 , image sensor 472 of smart glasses 1006 may capture an imageof a pointing gesture performed by user 1016. Processing device 460(FIG. 4 ) may analyze the image using a gesture recognition algorithmand identify the pointing gesture as a user input requesting to relocate(e.g., move) a specific virtual object (e.g., a particular virtualobject such as virtual house plant 1014) external to virtual display1002. In FIG. 11 , processing device 460 may respond to the user inputby causing virtual house plant 1014 of group 1050 of virtual objects tobe displayed at a new location in extended reality environment 1004,external to virtual display 1002, e.g., as though resting on desk top1020. In some implementations, processing device 460 may present virtualhouse plant 1014 inside virtual display 1002 concurrently withdisplaying version 1014A of virtual house plant 1014 external to virtualdisplay 1002.

In some embodiments the input includes at least one signal reflectingkeystrokes on a keyboard. The term “signal” may refer to a function thatcan vary over space and time to convey information observed about aphenomenon via a physical medium. For example, a signal may beimplemented in any range of the electromagnetic spectrum (e.g., radio,IR, optic), as an acoustic signal (e.g., audio, sonar, ultrasound), amechanical signal (e.g., pressure or vibration), as an electric ormagnetic signal, or any other type of signal. The phenomenoncommunicated by the signal may relate to a state, the presence orabsence of an object, an occurrence or development of an event oraction, or lack thereof. The term “reflecting” may refer to expressing,telling, or revealing a causality or consequence due to an action orstate (e.g., temporary, or steady state). The term “keystroke” may referto an action associated with selecting or operating a key of a physicalor virtual keyboard. The term “keyboard” may refer to an input deviceincluding multiple keys, each representing an alphanumeric character(letters and numbers), and optionally including a numeric keypad,special function keys, mouse cursor moving keys, and status lights, asdescribed earlier. A keystroke may be implemented by pressing a key of amechanical keyboard, by touching or swiping a key of a keyboarddisplayed on a touch-sensitive screen, by performing a typing gesture ona virtual or projected keyboard, or by any other technique for selectinga key. The at least one processor may receive a signal associated withthe input indicating (e.g., reflecting) one or more keystrokes performedby the user on a keyboard. In one example, the keystrokes may beindicative of a desire of the user to cause the specific virtual objectto move from the virtual display to the extended reality environment. Inone example, the keystrokes may be indicative of the specific virtualobject and/or of a desired position in the extended reality environmentfor the specific virtual object.

By way of a non-limiting example, in FIG. 11 , user 1016 may enter arequest to remove virtual house plant 1014 from virtual display 1002 byperforming one or more keystrokes on keyboard 1018 resting on desk top1020. Processing device 460 (FIG. 4 ) of smart glasses 1006 may receiveone or more signals associated with the keystrokes (e.g., via networkinterfaces 320 of FIG. 3 and 420 of FIG. 4 ) as a user input and respondto the input accordingly.

In some embodiments the input includes at least one signal reflecting amovement of a pointer. The term “movement” may refer to a motiondynamically changing a location, position and/or orientation, e.g., of avirtual or physical object. The term “pointer” may include technologyenabling the selection of content by targeting the selected content in afocused manner. A pointer may be an electronic pointing device or may beimplemented as a bodily gesture e.g., by the eye, head, finger, hand,foot, arm, leg, or any other moveable part of the body. Examples ofelectronic pointing devices may include an electronic mouse, stylus,pointing stick, or any other electronic pointing device. For example, aprocessing device may detect an IR signal of an electronic pointermaneuvered by a user. Alternatively, an image and/or motion sensor(e.g., image sensor 472 and/or motion sensor 373 of FIG. 4 ) may detecta pointing gesture by a user. In one example, the movement of thepointer may be indicative of a desire of the user to cause the specificvirtual object to move from the virtual display to the extended realityenvironment. In one example, the movement of the pointer may beindicative of the specific virtual object and/or of a desired positionin the extended reality environment for the specific virtual object.

As an example, a user may extend the index finger to target (e.g., pointto) a specific virtual object. Light reflecting off the index finger maybe captured (e.g., via image sensor 472 of FIG. 4 ) and stored as one ormore images. The at least one processor may analyze the one or moreimages to detect a pointing gesture by the index finger (e.g., movementof a pointer) in the direction of the virtual object. As anotherexample, a user may manipulate an IR pointer in the direction of avirtual display. An IR sensor (e.g., sensor 472) may detect IR light(e.g., a signal) emitted by the IR pointer and send a correspondingsignal to the at least one processor. The at least one processor mayanalyze the signal to determine the IR pointer targeting the specificvirtual object.

By way of a non-limiting example, in FIG. 11 , user 1016 may removevirtual house plant 1014 from virtual display 1002 by selecting anddragging virtual house plant 1014 to desk top 1020, external to virtualdisplay 1002. Processing device 460 (FIG. 4 ) of smart glasses 1006 maydetect the selecting and dragging operations of electronic mouse 1022(e.g., via network interfaces 320 of FIG. 3 and 420 of FIG. 4 ) anddisplay version 1014A of virtual house plant 1014 at the locationindicated by electronic mouse 1022.

Some embodiments further include analyzing the input from a pointingdevice to determine a cursor drag-and-drop movement of the specificvirtual object to a location outside the virtual display. The term“analyzing” may refer to investigating, scrutinizing and/or studying adata set, e.g., to determine a correlation, association, pattern, orlack thereof within the data set or with respect to a different dataset. The term “pointing device” may refer to a pointer implemented as anelectronic pointing device, as described earlier. The term “determine”may refer to performing a computation, or calculation to arrive at aconclusive or decisive outcome. The term “cursor drag-and-drop movement”may refer to interfacing with displayed content using a pointing deviceto control a cursor. A cursor may be a movable graphic indicator on adisplay identifying the point or object affected by user input. Forexample, a user may select a virtual object by controlling the pointingdevice to position the cursor on the virtual object and pushing a buttonof the pointing device. The user may move (e.g., drag) the selectedobject to a desired location by moving the pointing device while theobject is selected (e.g., while pressing the button). The user mayposition the selected object at the desired location by releasing (e.g.,dropping) the selection via the pointing device (e.g., by releasing thebutton). The combination of these actions may be interpreted by the atleast one processor as a cursor drag-and-drop movement. The pointingdevice may include one or more sensors to detect a push and/or releaseof a button of the pointing device, and one or more motion sensors todetect dragging a selected object to a desired location. An inputreceived from a pointing device may include electronic signals (e.g.,caused by pressing or releasing a button, or a motion of a roller ball),IR, ultrasound, radio (e.g., Bluetooth, Wi-Fi), IMU, and any other typeof signal indicating selection, dragging, and dropping by a pointingdevice. The one or more sensors may convert the input to an electronicsignal and provide the electronic signal to the at least one processor,as described earlier. The term “location” may refer to a position orregion, e.g., inside a larger area. A location may be relative to aphysical and/or virtual object. For example, the location outside thevirtual display may be relative to the wearable extended realityappliance, to an absolute coordinate system (e.g., GPS), to a physicalobject (e.g., a desk or wall), to virtual content such as the virtualdisplay, or to any other reference. In one example, the cursordrag-and-drop movement may be indicative of a desire of the user tocause the specific virtual object to move from the virtual display tothe extended reality environment. In one example, the cursordrag-and-drop movement may be indicative of the specific virtual objectand/or of the location outside the virtual display.

As an example, a user may maneuver a pointing device to move a specificvirtual object to a different location external to the virtual display.The user may control the pointing device to position the cursor on avirtual object, press a button of the pointing device to select thevirtual object, move the pointing device while pressing the button toreposition the selected object to a new location external to the virtualdisplay, and release the pressed button to drop the virtual object atthe new location. The pointing device may provide inputs to the at leastone processor indicative of the cursor position, button press, draggingmotion, and button release. The at least one processor may analyze theinputs to determine a cursor drag-and-drop movement by the pointingdevice relocating the virtual object to the new location, e.g., outsidethe virtual display. In response, the processing device may display thevirtual object at the new location.

By way of a non-limiting example, in FIGS. 10 and 11 , user 1016 maymaneuver electronic mouse 1022 to move virtual house plant 1014 to alocation external to virtual display 1002. For example, user 1016 mayuse electronic mouse 1022 to maneuver a cursor over virtual house plant1014 and push a button of electronic mouse 1022 to turn the focusthereon. While the focus is on virtual house plant 1014, user 1016 maymove (e.g., drag) electronic mouse 1022 to cause a correspondingmovement by virtual house plant 1014. When virtual house plant 1014 ispositioned on desk top 1020, e.g., external to virtual display 1002,user 1016 may release the button to position (e.g., drop) virtual houseplant 1014 on desk top 1020. Throughout the maneuvering by user 1016,electronic mouse 1022 may provide signals indicating any movements,button presses and releases as inputs to processing device 460 (FIG. 4), e.g., as pointer input 331 via input interface 330. Processing device460 may analyze the inputs and determine a cursor drag-and-drop movementof electronic mouse 1022 corresponding to a repositioning virtual houseplant 1014 from inside virtual display 1002 (e.g., as shown in FIG. 10 )to desk top 1020, external to virtual display 1002 (e.g., as shown inFIG. 11 ), and may update the position of virtual house plant 1014,accordingly.

Some embodiments further include analyzing movement of the pointer todetermine a selection of an option in a menu bar associated with thespecific virtual object. The term “selection” may refer to picking orchoosing an object, e.g., from one or more objects. Selecting an objectvia a pointing device may turn the focus on the selected object suchthat subsequent input affects the selected object. The term “menu bar”may refer to a graphical control element including one or moreselectable items, values, or other graphical widgets (e.g., buttons,checkboxes, list boxes, drop down lists, and pull-down lists). Forexample, one menu may provide access to functions for interfacing with acomputing device and another menu may be used to control the display ofcontent. A menu bar may include multiple drop-down menus that normallyhide the list of items contained in the menu. Selecting a menu (e.g.,using the pointer) may display the list of items. The term “option in amenu bar” may refer to a specific menu item displayed in the menu bar.The term “associated with” may refer to linked or affiliated with ortied or related to. In one example, the option in the menu bar may beindicative of a desire of the user to cause the specific virtual objectto move from the virtual display to the extended reality environment. Inone example, the option in the menu bar may be indicative of a desiredposition in the extended reality environment for the specific virtualobject.

As an example, a user may use a pointer to select an option on a menubar associated with a specific virtual object, (e.g., as pointer input331 via input interface 330 of FIG. 3 ). The option may allow alteringthe display of the virtual object (e.g., to enlarge, shrink, move, hide,or otherwise change the display of the virtual object). The at least oneprocessor may analyze the movements of the pointer to detect theselection of the option and execute a corresponding action.

Some embodiments involve in response to receiving the input, generatinga presentation of a version of the specific virtual object in theextended reality environment at a third virtual distance from thewearable extended reality appliance, wherein the third virtual distancediffers from the first virtual distance and the second virtual distance.The term “presentation of a version of the specific virtual object” mayrefer to another rendition or depiction of the specific virtual object.The version of the specific virtual object may be presented alongside orto replace the specific virtual object. The term “differs” may refer tobeing distinguished or distinct from, or otherwise dissimilar. The term“third virtual distance” may be interpreted in a manner similar to theinterpretation of first distance and second distance describe earlier.

For example, the version of the specific virtual object may be displayed(e.g., presented) to appear identical to, similar to, or different fromthe specific virtual object, e.g., the version of the specific virtualobject may be a smaller or larger replica of the specific virtualobject. As another example, the version of the specific virtual objectmay appear identical or similar to the specific virtual object but maybe displayed in a different location, e.g., the specific virtual objectmay be displayed inside a virtual display, whereas the version of thespecific object may appear identical but may be displayed external tothe virtual display. As another example, the orientations or angulardistances of the specific virtual object and the version of the specificvirtual object relative to the user and/or the wearable extended realityappliance may be the same or different. As another example, the versionof the virtual object may be presented inside the field-of-view of thewearable extended reality appliance, outside the field-of-view, orpartially inside and partially outside the field-of-view. As a furtherexample, the version of the specific virtual object may be rendereddifferently, e.g., using different colors, resolution, or a differentcoordinate system, e.g., the specific virtual object may be displayed asa two-dimensional (e.g., simplified) object, and the version of thespecific virtual object may be presented as a three-dimensionallife-like object. Thus, upon receiving an input to move a widget (e.g.,specific virtual object) from the virtual display), the wearableextended reality appliance may generate a version of the widget anddisplay the version at a virtual distance different than the virtualdistance to the virtual display and to the additional virtual object.

By way of a non-limiting example, In FIG. 10 , smart glasses 1006 mayvisually present virtual display 1002 and virtual mobile phone 1026 atvirtual distances D1 and D2, respectively, from user 1016 (e.g.,measured with respect to 3D coordinate system 1028). User 1016 mayperform a gesture corresponding to a request to move virtual house plant1014 to a location in extended reality environment 1004, external tovirtual display 1002. The gesture may be detected via image sensor 474(FIG. 4 ). Processing device 460 may analyze image data acquired byimage sensor 474 to identify the gesture as a user input. In FIG. 11 ,in response to the user input, processing device 460 may obtain aversion 1014A of virtual house plant 1014 (e.g., by retrieving version1014A from memory device 411). Processing device 460 may display version1014A to appear as though resting on desk top 1020 at a distance D3 fromuser 1016, where D3 differs from D1 and D2 (e.g., with respect to 3Dcoordinate system 1028). While the example of FIG. 11 shows version1014A replacing virtual house plant 1014, in some implementations,version 1014A may be displayed alongside (e.g., concurrently with)virtual house plant 1014. In some implementations, version 1014A may bedisplayed differently (e.g., larger/smaller, higher/lower resolution,modified color scheme) than virtual house plant 1014, e.g., virtualhouse plant 1014 may be rendered as a 2D graphic image, and version1014A may be rendered as a 3D graphic image.

In some embodiments, when the specific virtual object is a windowincluding a group of control buttons in a particular area of the window,the group of control buttons include at least a control button forminimizing the window, a control button for maximizing the window, acontrol button for closing the window, and a control button for movingthe window outside the virtual display; and wherein the input includesan activation of the control button for moving the window outside thevirtual display. The term “window” may refer to a graphic controlelement (e.g., 2D or 3D) providing a separate viewing area on a displayscreen. A window may provide a single viewing area, and multiple windowsmay each provide a different viewing area. A window may be part of agraphical user interface (GUI) allowing users to input and view outputand may include control elements, such as a menu bar along the topborder. A window may be associated with a specific application (e.g.,text editor, spread sheet, image editor) and may overlap or be displayedalongside other windows associated with the same, or differentapplications. A window may be resized (e.g., widened, narrowed,lengthened, or shortened), opened (e.g., by double clicking on an iconor menu item associated with the window), or closed (e.g., by selectingan “X” control element displayed at a corner of the window. The term“control button” may refer to a graphic element that invokes an actionupon selection (e.g., via a pointing device, keystroke, or gesture). Forexample, an operating system may receive a notification when the userselects a control button and may schedule a processing device to executea corresponding action. The term “group of control buttons” may refer toa collection of one or more control buttons. The term “particular areaof the window” may refer to a specific region within a window graphiccontrol element, e.g., the group of control buttons may be located in aspecific region of a window, such as across the top, or along a side asa menu bar. The term “minimizing the window” may refer to collapsing thewindow such to hide the window from view while allowing an applicationassociated with the window to continue running. A minimized window mayappear at the bottom of a display as an icon inside a task bar. The term“maximizing the window” may refer to expanding the window to occupy someor all of the display screen. The term “closing the window” may refer toremoving the window from a display screen and halting the execution ofthe associated application. The term “moving the window” may refer tochanging the position of a window in a display screen. As an example, awindow may be dragged up/down, right/left or diagonally across atwo-dimensional display. In a 3D-environment, such as an extendedreality environment generated by a wearable extended reality appliance,a window may be additionally or alternatively dragged inwards/outwards.The term “activation of the control button for moving the window outsidethe virtual display” may be understood as selecting the control buttonto invoke an action that relocates the window external to the virtualdisplay.

As an example, a virtual display may include a window containing avirtual document. The window may be sized to fit inside the virtualdisplay alongside other virtual objects and may include a menu bar withcontrol buttons to minimize, maximize, close, and move the window. Auser wishing to read and edit the virtual document may select thecontrol button to move the window out of the virtual display, to displaythe window closer and larger (e.g., using a larger font size).

By way of a non-limiting example, reference is made to FIG. 12 , whichillustrates the exemplary environment of FIGS. 10 and 11 (e.g.,generated by system 200) where the content includes a window having acontrol button for moving content between a virtual display and anextended reality environment, consistent with some embodiments of thepresent disclosure. FIG. 12 is substantially similar to FIGS. 10 and 11with the noted difference that virtual display 1002 may present a window1200 associated with a text editing application. Window 1200 may includea group 1202 of control buttons at the top region of window 1200. Fromright to left, group 1202 of control buttons may include buttons forclosing, maximizing, and minimizing, window 1200, and additionally acontrol button 1204 for moving window 1200 outside of virtual display1002. User 1016 may activate control button 1204 by performing apointing gesture (e.g., captured as image data via image sensor 472 ofFIG. 4 ), using a mouse cursor, using a keyboard, and so forth. Inresponse, processing device 460 may display version 1200A of window 1200external to virtual display 1002.

Some embodiments involve causing, in response to receiving the input, apresentation of the specific virtual object to be removed from thevirtual display. The term “removed” may refer to eliminated or erased.Thus, in response to the input, the wearable extended reality appliancemay display a version (e.g., copy) of the specific object in a regionexternal to the virtual display and may remove the presentation of thevirtual object from inside the virtual display.

By way of a non-limiting example, in FIG. 11 , in response to receivingan input from user 1016, processing device 460 (FIG. 4 ) may displayversion 1014A of virtual house plant 1014 (FIG. 10 ) to appear to desktop 1020 (e.g., external to virtual display 1002), and remove virtualhouse plant 1014 from being displayed inside virtual display 1002.

Some embodiments involve causing, in response to receiving the input,simultaneous presentations of the specific virtual object on the virtualdisplay and the version of the specific virtual object at anotherlocation in the extended reality environment. The term “simultaneous”may refer to concurrent, or at substantially the same time. The term“another location” may refer to a separate location, e.g., differentfrom an original location. Thus, in response to the input, the wearableextended reality appliance may display a version (e.g., copy) of thespecific object in a region external to the virtual display andconcurrently with displaying the virtual object inside the virtualdisplay.

By way of a non-limiting example, reference is made to FIG. 13 whichillustrates the exemplary environment of FIGS. 10 and 11 where aspecific virtual object is displayed inside a virtual displayconcurrently with a version of the specific object displayed external tothe virtual display, consistent with some embodiments of the presentdisclosure. In response to receiving a gesture input from user 1016 todisplay virtual house plant 1014 external to virtual display 1002,processing device 460 (FIG. 4 ) may display version 1014A to appear asthough resting on desk top 1020 at a distance D3 from smart glasses1006, concurrently with displaying virtual house plant 1014 insidevirtual display 1002 at a distance D1 from smart glasses 1006. As anexample, virtual house plant 1014 may be a two-dimensional icon, andversion 1014A may be a realistic three-dimensional rendition of a houseplant.

Some embodiments involve determining the third virtual distance forpresenting the version of the specific virtual object. The at least oneprocessor may determine the third virtual distance based on, forexample, criteria relating to the extended reality environment (e.g.,virtual and/or physical considerations), criteria relating to thewearable extended reality appliance (e.g., device considerations),criteria relating to the communications network (e.g., bandwidthconsiderations), or any other criteria. For example, ambient light, andthe presence of obstructing objects may be relevant for determining thethird virtual distance. As another example, the type of virtual objectmay be used to determine the third virtual distance, e.g., a textdocument may be displayed closer to allow editing, and a decorativevirtual object may be displayed further.

By way of a non-limiting example, in FIG. 11 , processing device 460(FIG. 4 ) may determine virtual distance D3 for presenting version 1014Aof virtual house plant 1014. The virtual distance may be based on thedistance between user 1016 and desk top 1020.

In some embodiments, the determination of the third virtual distance isbased on at least one of the first virtual distance or the secondvirtual distance. The term “based on” may refer to established orfounded upon, or otherwise derived from. For example, the at least oneprocessor may determine the third virtual distance (e.g., the distancefor displaying the version of the virtual object extracted from thevirtual display) based on one or more of the other distances to thevirtual display (e.g., the first virtual distance) and the additionalvirtual object (e.g., the second virtual distance). The third virtualdistance may be determined to avoid obstruction by the virtual displayand/or the additional virtual object. The determined third virtualdistance may be greater or smaller than one or both the first and secondvirtual distances, or a combination (e.g., Euclidian distance) of thefirst and second virtual distances. As an example, the third virtualdistance may have the same height along the vertical plane but maydiffer along the horizontal plane. In some examples, the third virtualdistance may be a mathematical function of the first virtual distanceand/or the second virtual distance. For example, the mathematicalfunction may be a linear function, may be a non-linear function, may bea polynomial function, may be an exponential function, may be amultivariate function, and so forth.

By way of a non-limiting example, in FIG. 11 , processing device 460(FIG. 4 ) of smart glasses 1006 may determine virtual distance D3 fordisplaying version 1014A of virtual house plant 1014 based on virtualdistance D1 between virtual display 1002 and user 1016, and/or based onvirtual distance D2 between virtual mobile phone 1026 and user 1016.Processing device 460 may determine virtual distance D3 so that virtualhouse plant 1014A is not obstructed by virtual mobile phone 1026 and/orvirtual display 1002.

In some embodiments, the determination of the third virtual distance isbased on a type of the specific virtual object. The term “type of thespecific virtual object” may refer to a category or classification ofthe specific virtual object. For example, a virtual object may beclassified according to data type (e.g., text, image, video), data size(e.g., related to communications bandwidth, processing, and/or memoryrequirements), spatial size, whether the specific virtual object is 2Dor 3D, whether the specific virtual object is interactive, transparency(e.g., displayed as semi-transparent or opaque), use (e.g., read-only,or editable), priority (e.g., urgent messages or work-related documentsversus low priority ornamental objects), security (e.g., proprietary orprivileged access), or any other criterion for determining a type for avirtual object.

By way of a non-limiting example, in FIG. 11 , processing device 460(FIG. 4 ) may determine virtual distance D3 to version 1014A of virtualhouse plant 1014 based on virtual house plant 1014 being a decorativevirtual object. Thus, D3 may be determined to be further from user 1016than distance D2 to virtual display 1002.

In some embodiments, the determination of the third virtual distance isbased on a physical object in the extended reality environment. The term“physical object in the extended reality environment” may refer to areal (e.g., tangible) article or item. For example, a physical objectmay be a tangible (e.g., real) bookcase, wall, or floor, a light source(e.g., a window, or light fixture), a person or animal. The physicalobject may be stationary or in motion, all or partially opaque, ortransparent. The physical object may be detected by analyzing dataacquired via a sensor (e.g., via sensors interface 470 of FIG. 4 ). Forexample, image or IR data may be acquired via an image sensor (e.g.,image sensor 472 and/or 372 of FIG. 3 ), motion data may be acquired viaa motion sensor (e.g., motion sensor 473 and/or 373), ultrasound and/orother data may be acquired via other sensors (e.g., other sensors 475and/or 375). For example, image data captured using at least one imagesensor may be analyzed using an object detection algorithm to detect thephysical object. Thus, the wearable extended reality appliance maydetermine the distance for displaying the specific virtual object basedon one or more physical items present in the extended realityenvironment, e.g., to prevent obstruction. As another example, thephysical object may be used to scale the virtual object (e.g., to appearcloser or further) from the wearable extended reality appliance. In someexamples, the determination of the at third virtual distance may bebased on at least one of a distance to the physical object, a positionof the physical object, a size of the physical object, a color of thephysical object, or a type of the physical object. In one example, thephysical object may include a surface (such as a table including a tabletop surface), and the third virtual distance may be select to positionthe version of the specific virtual object on a central portion of thesurface. In one example, the third virtual distance may be selected tobe shorter than a distance to the physical object, for example to makethe specific virtual object hide at least part of the physical object.In one example, the third virtual distance may be selected to be longerthan a distance to the physical object, for example to make at leastpart of the specific virtual object hidden by the physical object. Inone example, the third virtual distance may be selected to be similar toa distance to the physical object, for example to make the specificvirtual object appear side by side with the physical object.

By way of a non-limiting example, in FIG. 11 , processing device 460(FIG. 4 ) of smart glasses 1006 may detect physical desk top 1020 insideextended reality environment 1004. Upon receiving an input to extractvirtual house plant 1014 from virtual display 1002, processing device460 may determine to display version 1014A of virtual house plant 1014to appear as though resting on desk top 1020. Processing device 460 maydetermine virtual distance D3 separating version 1014A from smartglasses 1006 based on the location of desk top 1020.

Some embodiments involve determining a position for presenting theversion of the specific virtual object in the extended realityenvironment. The term “position” (e.g., for an object in the extendedreality environment) may refer to a distance (e.g., relative to aphysical and/or virtual object with respect to a 2D or 3D coordinatesystem) and/or an orientation, bearing, or pose of the object. Forexample, the position may determine where in the 3D space to display anobject, as well as an angular orientation for the object (e.g., turnedbackwards, upside-down, rotated by an angle). Thus, in response toreceiving an input to remove a virtual object from the virtual display,the at least one processor may determine the location and/ororientation, pose, or bearing for the version of the virtual objectoutside the virtual display. The position may be determined based onother objects (virtual and/or real) in the extended reality environment,on environmental conditions (e.g., ambient light, noise, or wind), onthe size and/or shape of the extended reality environment, and any othercriterion for determining a position for presenting virtual content. Forexample, the processor may determine a position of the virtual object infront of, behind, or to one side of another virtual or real object inthe extended reality environment.

By way of a non-limiting example, in FIG. 11 , processing device 460(FIG. 4 ) may determine a position in extended reality environment 1004for presenting version 1014A of virtual house plant 1014. Processingdevice 460 may determine the distance D3 between version 1014A ofvirtual house plant 1014 and smart glasses 1006 as a three-dimensionaldiagonal with respect to 3D coordinate system 1028. Processing device460 may additionally determine an orientation for version 1014A ofvirtual house plant 1014, e.g., with respect to one or more axes of 3Dcoordinate system 1028. Processing device 460 may determine to orientversion 1014A of virtual house plant 1014 such that the pot portion isbelow the leaves portion (e.g., rotation about the x-axis and z-axis),and to present the largest leaves in the direction of user 1016 (e.g.,rotation about the y-axis).

In some embodiments, the determination of the position is based on aphysical object in the extended reality environment. For example, thewearable extended reality appliance may determine the position for theversion of the specific virtual object based on one or more physicalitems, e.g., to avoid obstruction by the physical items and/or tointegrate the specific virtual object with the physical environmentsurrounding the user. As another example, the wearable extended realityappliance may determine an angular position or orientation forpresenting the version of the specific virtual object in the extendedreality environment, e.g., relative to the wearable extended realityappliance and/or the user. In one example, the position for presentingthe version of the specific virtual object may be determined based on athird virtual distance, and the third virtual distance may be determinedas described earlier. In one example, image data captured using at leastone image sensor and/or 3D data captured using Lidar may be analyzedusing an object localization algorithm to detect a position of thephysical object. Further, the position for presenting the version of thespecific virtual object may be a mathematical function of the positionof the physical object. In one example, image data captured using atleast one image sensor may be analyzed using an object recognitionalgorithm to identify a type of the physical object. Further, theposition for presenting the version of the specific virtual object maybe determined based on the type of the physical object.

By way of a non-limiting example, in FIG. 11 , processing device 460(FIG. 4 ) may detect the presence of physical desk top 1020 in extendedreality environment 1004 (e.g., via image sensor 472). Processing device460 may determine that desk top 1020 may provide a suitable surface fordisplaying version 1014A of virtual house plant 1014 in a manner thatintegrates virtual content with the real (e.g., physical) environmentsurrounding user 1016. Processing device 460 may determine the positionfor displaying version 1014A of virtual house plant 1014 to appear asthough resting on desk top 1020 without colliding with keyboard 1018 andelectronic mouse 1022 (e.g., physical objects).

In some embodiments, generating the presentation of the version of thespecific virtual object in the extended reality environment includesimplementing a modification to the specific virtual object. The term“implementing a modification” may refer to adjusting or changing one ormore attributes of the virtual object, e.g., for presenting the specificvirtual object. As an example, the modification may include adding orremoving an audio presentation (e.g., accompanying sound) for adisplayed object, or replacing a displayed object with an audiblepresentation. As another example, the modification may include adding orremoving a haptic response associated with interfacing with the virtualobject. As another example, the adjusted attributes may affect thedisplay or appearance of the version of the specific virtual object. Forexample, such adjusted attributes may include one or more of size,magnification, texture, color, contrast, and other attributes associatedwith an appearance of the virtual object in the extended realityenvironment. In yet another example, the modification may includemodifying a 2D specific virtual object to an associated 3D virtualobject. In an addition example, the modification may include modifyingat least one of a color scheme, a size or an opacity associated with thespecific virtual object. In another example, the modification mayinclude expanding unexpanded elements of the specific virtual object.For example, the specific virtual object may include a plurality ofmenus or sections. While in the virtual display, expanding the menus orsections may create clutter due to the limited size within the virtualdisplay. When the specific virtual object is displayed outside thevirtual display, the specific virtual object may spread over a largerarea, and thus the menus or sections may be expanded without creatingclutter.

By way of a non-limiting example, in FIG. 10 , virtual house plant 1014(e.g., the specific virtual object) is displayed inside virtual display1002 as a simplified two-dimensional drawing or icon. Upon receiving aninput to extract virtual house plant 1014 from virtual display 1002,processing device 460 (FIG. 4 ) may modify the appearance of virtualhouse plant 1014, e.g., by retrieving from a memory (e.g., datastructure 212 of FIG. 2 ) version 1014A of virtual house plant 1014,which may be a 3D rendition of a physical house plant generated, forexample, by combining multiple high-resolution 2D images of the physicalhouse plant.

In some embodiments, the modification to the specific virtual object isbased on a type of the specific virtual object. For example, thewearable extended reality appliance may modify how the specific virtualobject is presented outside the virtual display based on the objecttype. Examples of object types include, for example, documents, icons,charts, graphical representations, images, videos, animations, chatbots, or any other category or sub-category of visual display. Theobject type may indicate how the user wishes to consume or use thecontent, e.g., a document intended for editing may be enlarged whenextracted from the virtual display, and an object displayed inside thevirtual display in 2D may be converted to a 3D rendition outside thevirtual display. Other examples may include displaying a messaging appas an icon inside the virtual display and adding multiple controlelements outside the virtual display, displaying a graphic image usinglow saturation inside the virtual display and with a higher saturationoutside the virtual display, presenting a music widget visually insidethe virtual display and audibly outside the virtual display, presentinga video widget as a stationary image inside the virtual display and asan animated video outside the virtual display.

By way of a non-limiting example, in FIG. 11 , processing device 460(FIG. 4 ) may determine, based on virtual house plant 1014 of FIG. 10being an ornamental object, to modify the appearance when displayedoutside of virtual display 1002 to a 3D version 1014A of a house plant.As another example, in FIG. 12 , upon detecting an input to move window1200 from virtual display 1002, processing device 460 may determine thatwindow 1200 includes an editable document and may increase the size ofversion 1200A of window 1200.

In some embodiments, the modification to the specific virtual objectincludes changing a visual appearance of the specific virtual object.The term “changing a visual appearance of the specific virtual object”may refer to adjusting a display characteristic of the virtual object.For example, the modification may change a color, color scheme (e.g.,black/white, grey scale, or full color gamut), resolution, sire,scaling, transparency, opacity, saturation, intensity, or any otherdisplay characteristic of the virtual object. As another example, themodification may convert a 2D image to a 3D rendition, or vice-versa. Asanother example, the specific virtual object (e.g., displayed inside thevirtual display) may be a simplified representation of the virtualobject (e.g., such as an icon or 2D drawing) and the version of thespecific virtual object displayed external to the virtual display mayinclude additional details and resolution (e.g., based on one or morehigh resolution images acquired of a physical object representative ofthe icon or 2D drawing).

By way of a non-limiting example, in FIG. 11 , to generate version 1014Aof virtual house plant 1014 (e.g., shown as a 2D drawing inside virtualdisplay 1002), processing device 460 (FIG. 4 ) may convert the 2Ddrawing to 3D version 1014A.

In some embodiments, the modification to the specific virtual objectincludes changing a behavior of the specific virtual object. The term“changing a behavior of the specific virtual object” may refer tomodifying or adjusting interactive aspects of the specific virtualobject, e.g., with respect to a user, a device, a software application,or any other entity interfacing with the virtual object. For example, agraphic image may be displayed as a static image inside the virtualdisplay, and as an animated image (e.g., GIF) outside the virtualdisplay. As another example, a text document may be read-only inside thevirtual display and may be editable outside the virtual display. Asanother example, control elements such as buttons and text boxes may beinactive inside the virtual display and active when displayed externalto the virtual display. As another example, when the specific virtualobject is presented in the virtual display, menus and/or section of thespecific virtual object may be automatically minimized when not in use(for example to minimize clutter, as described earlier), and when thespecific virtual object is presented outside the virtual display, themenus and/or sections may be displayed in an expanded form, when not inuse.

By way of a non-limiting example, in FIG. 12 , window 1200 may beread-only while displayed inside virtual display 1002. Upon receiving aninput from user 1016 to move window 1200 out from virtual display 1002(e.g., by selecting control button 1204), processing device 460 maydisplay version 1200A external to virtual display 1002. Version 1200A ofwindow 1200 may be editable by user 1016, e.g., using a pointing deviceor hand gestures.

Some embodiments involve receiving an additional input for causinganother virtual object from the group of virtual objects to move fromthe virtual display to the extended reality environment; and in responseto receiving the additional input, generating a presentation of aversion of the another virtual object in the extended realityenvironment at a fourth virtual distance from the wearable extendedreality appliance, wherein the fourth virtual distance differs from thefirst virtual distance, the second virtual distance, and the thirdvirtual distance. The term “additional input” may refer to a separateinput, e.g., different than the input associated with the specificvirtual object (e.g., the first input). The inputs may be via the sameor different medium. For example, a gesture input may be used to move afirst virtual object (e.g., the specific virtual object) from thevirtual display and an electronic pointer or voice command may be usedto move a second virtual object. The additional input may be received atthe same or different time (e.g., after) the first input, and/or may beassociated with a different virtual object or group of objects. The term“in response to receiving the additional input” may refer to in reactionto, or consequent to receiving the additional input. The term “fourthdistance” may be interpreted in a manner similar to the interpretationof first distance, second, and third distance describe earlier. Thus,the at least one processor may be responsive to multiple differentinputs for manipulating and/or modifying the presentation of the same ordifferent virtual objects in the extended reality environment and mayrespond to the different inputs accordingly. As an example, the at leastone processor may respond to the additional input by moving a virtualobject targeted by the additional input from the virtual display to adifferent location in the extended reality environment, e.g., that doesnot collide or overlap with another virtual object.

By way of a non-limiting example, reference is now made to FIG. 14 ,which illustrates the exemplary system of FIGS. 10 and 11 where anadditional virtual object included inside the virtual display is movedexternal to the virtual display, consistent with some embodiments of thepresent disclosure. FIG. 15 is substantially similar to FIG. 11 with thenoted difference that, in addition to version 1014A of virtual houseplant 1014 located external to virtual display 1002, virtual workspace1012 is moved outside of virtual display 1002, (e.g., indicated asversion 1012A of virtual workspace 1012). After performing a pointinggesture to move virtual house plant 1014 out from virtual display 1002,user 1016 may use a voice command to move virtual workspace 1012 out ofvirtual display 1002. The voice command may be detected by a microphone(e.g., audio sensor 472 of FIG. 4 ) configured with smart glasses 1006.In response, processing device 460 may determine a distance D4 (e.g., afourth virtual distance) for displaying version 1012A of virtualworkspace 1012 smaller than distances D1, D2, or D3, e.g., to facilitatereading.

Some embodiments involve identifying a trigger for halting thepresentation of the version of the specific virtual object in theextended reality environment and for presenting the specific virtualobject on the virtual display. The term “identifying” may refer torecognizing, perceiving, or otherwise determining or establishing anassociation. A processing device may identify a type of input by parsingthe input and performing one or more comparisons, queries, or inferenceoperations. The term “trigger” may refer to an event, occurrence,condition, or rule outcome that when provokes, causes, or promptssomething, in this instance the halting of the presentation. Forexample, a warning signal that a device is overheating may trigger thedevice to shut down. The term “halting” may refer to pausing, delaying,ceasing, or terminating, e.g., an execution of an application. Thus, theat least one processor may identify a received input as a prompt (e.g.,trigger) to cease presenting the version of the specific virtual objectexternal to the virtual display and revert to presenting the specificvirtual object inside the virtual display. The input may be provided asa gesture, via a pointing device or keyboard, as a voice command, or anyother user interfacing medium. As an example, the user may drag theversion of the specific virtual object back into the virtual display,delete or close the version of the specific virtual object, or performany other operation to cease presenting the version of the specificvirtual object outside the virtual display. In some examples, thetrigger may be or include a person approaching the user. For example, aperson approaching the user may be identified as described earlier. Insome examples, the trigger may be or may include an interaction betweenthe user and another person (e.g., a conversation). For example, audiodata captured using an audio sensor included in the wearable extendedreality appliance may be analyzed using speech recognition algorithms toidentify a conversation of the user with another person.

By way of a non-limiting example, reference is made to FIG. 15 , whichillustrates the exemplary system of FIGS. 10 and 11 where a trigger isidentified for halting a presentation of content external to the virtualdisplay, consistent with some embodiments of the present disclosure.FIG. 15 is substantially similar to FIG. 11 with the noted differencethat user 1016 is pointing to version 1014A of virtual house plant 1014.Processing device 460 (FIG. 4 ) may identify the point gesture as atrigger to halt presenting version 1014A of virtual house plant 1014 ondesk top 1020 (e.g., in extended reality environment 1004), and torevert presenting virtual house plant 1014 inside virtual display 1002,e.g., as illustrated in FIG. 10 .

In some embodiments, the trigger includes at least one of: an additionalinput, a change in operational status of the wearable extended realityappliance, or an event associated with predefined rules. For example,the additional input may originate from the user, from the wearableextended reality appliance, or from another computing device (e.g., aperipheral device, server, mobile phone). The term “operational statusof the wearable extended reality appliance” may refer to a functioningor working state of the wearable extended reality appliance. Examples ofoperational status of the wearable extended reality appliance mayinclude a battery level, an amount of available memory, computation orcommunications capacity, latency (e.g., for communications, processing,reading/writing), a temperature (e.g., indicating overheating of anelectronic component), a setting of a switch (e.g., hardware and/orsoftware switch) relating to the operation of the wearable extendedreality appliance, and any other parameter affecting the operation ofthe wearable extended reality appliance. The term “change in operationalstatus” may refer to a development that alters the functioning of thewearable extended reality appliance, e.g., a battery level may becomelow, electronic components may overheat, memory buffers andcommunications channels may overflow.

The term “predefined rules” may refer to a set of guidelines,regulations, or directives specified in advance, e.g., to govern theoperation of the wearable extended reality appliance. Predefined rulesmay include general rules, and/or rules defined for a specific device,user, system, time, and/or context. The predefined rules may be storedin a database on a memory device (e.g., local and/or remote) andaccessed via query. The memory device may be accessible only forprivileged users (e.g., based on a device and/or user ID) or generallyaccessible. For example, one rule may increase the display intensitywhen ambient light exceeds a threshold, another rule may reorganize thedisplay of content when the number of displayed virtual objects exceedsa threshold, and a third rule may invoke a second application inreaction to invoking a first application. The term “event associatedwith predefined rules” may refer to data or signal that is generated orreceived indicating a circumstance affiliated with one or morepredefined rules. Examples of events may include an ambient lightwarning, a clutter event indicating the number of displayed virtualobjects exceeds a threshold, a user input to move an object, or anyother occurrence triggering a corresponding action. The at least oneprocessor may handle each event in compliance with an associated rule.As an example, an ambient light warning may cause content to bedisplayed using an increased intensity, a clutter event may cause somecontent to be minimized, and a user input may cause the object to bedisplayed in a different location.

Thus, the trigger received by the at least one processor (e.g., to haltthe presentation of the version of the specific virtual object) mayinclude one or more of an input, a notification indicating a change inthe operating state of the wearable extended reality appliance, or anevent affiliated with one or more rules. For example, the trigger may bea user pointing to close a virtual object, a voice command to move avirtual object back to the virtual display (e.g., additional inputs), atimeout event relating to the execution of a procedure (e.g., a changein operational status), or a clutter warning that the displayed virtualcontent exceeds a threshold.

By way of a non-limiting example, in FIG. 15 , processing device (460)may receive a pointing gesture (e.g., additional input) from user 1016,and may identify the pointing gesture as a trigger to halt thepresentation of version 1014A of virtual house plant 1014 on desk top1020). As another example, processing device 460 may receive a latencywarning from network interface 420 as a trigger to halt the presentationof version 1014A. As another example, processing device 460 may receivean ambient light warning from image sensor 472 as a trigger to halt thepresentation of version 1014A.

Some embodiments involve, after generating the presentation of theversion of the specific virtual object in the extended realityenvironment: while a focus of an operating system controlling the groupof virtual objects is a particular virtual object presented in thevirtual display, receiving a first input for task switching from akeyboard; and in response to receiving the first input for taskswitching, causing the focus of the operating system to switch from theparticular virtual object presented in the virtual display to theversion of the specific virtual object presented in the extended realityenvironment; while the focus of the operating system is the version ofthe specific virtual object presented in the extended realityenvironment, receiving a second input for task switching from akeyboard; and in response to receiving the second input for taskswitching, causing the focus of the operating system to switch from theversion of the specific virtual object presented in the extended realityenvironment to another virtual object presented in the virtual display.

The term “operating system” may refer to system software governinghardware and/or software resources of a computing device and providingservices, such as resource allocation, task scheduling, memory storageand retrieval, and other administrative services. For example, awearable extended reality appliance may be configured with an operatingsystem to manage system resources needed to generate the extendedreality environment. Referring to FIG. 4 , the operating system mayallocate space in memory device 411, schedule processing time forprocessing device 460, schedule the sending and receiving of data vianetwork interface 420, manage event listeners for receivingnotifications via sensors interface 470, output interface 450, and inputinterface 430, and perform additional task needed by the wearableextended reality appliance to generate the extended reality environment.The term “operating system controlling the group of virtual objects” mayrefer to the operating system administering computing resources of acomputing device, such a wearable extended reality appliance, forpresenting virtual objects. The term “focus of an operating system” mayrefer to an element in a graphical user interface that is currentlydesignated by the operating system as active. The operating system mayallocate resources (e.g., stack, queue, and buffer memory) and scheduleprocessing time such that user inputs received while a specificgraphical element is in focus affect the specific graphical element. Forexample, if a user inputs a move, maximize, or minimize instructionwhile a specific graphical element in focus, the move, maximize orminimize instruction may be implemented with respect to that specificgraphical element. In another example, if a user enters text (forexample, through voice, through a virtual keyboard, through a physicalkeyboard, etc.) while a specific application is in focus, the text maybe directed to the specific application. During an extended realitysession, the focus of the operating system may switch between differentvirtual objects, some presented inside the virtual display, and somepresented external to the virtual display. While the focus of theoperating system is on a particular virtual object inside the virtualdisplay, the operating system may allocate system resources for theparticular virtual object such that received inputs are implemented withrespect to that particular virtual object.

The term “task switching” may refer to swapping a currently executedprocess with a different process. An operating system may suspend acurrently executed process (e.g., task) by removing (e.g., popping) theprocess from a call stack and storing state data for the suspendedprocess. The state data may allow to subsequently restore the executionof the suspended process from the point of suspension. The operatingsystem may initiate the execution of the different process by retrievingstate data for the different process, and adding (e.g., pushing) thedifferent process onto the call stack. Task switching may be invokedautomatically (e.g., determined internally by the operating system),based on a user input, e.g., via voice command, pointing device,gesture, keyboard, or any other input means, based on an external input(e.g., from a peripheral device), or any other technique for invokingtask switching. Thus, for example, “receiving a first input for taskswitching from a keyboard” may refer to receiving one or more keystrokeinputs (e.g., “Alt+Tab”) via a keyboard requesting to switch to adifferent task. The operating system may identify the input as a requestfor task switching and may schedule the new task accordingly.

The term “causing the focus of the operating system to switch from theparticular virtual object presented in the virtual display to theversion of the specific virtual object presented in the extended realityenvironment” may be understood as implementing a task switching thattransfers the focus of the operating system from the particular virtualobject inside the virtual display onto the specific virtual objectexternal to the virtual display. Subsequent inputs may be implementedwith respect to the version of the specific virtual object external tothe virtual display. Thus, while the version of the specific objectexternal to the virtual display is in focus, the operating system mayallocate system resources such that received inputs are implemented withrespect to the version of the specific object.

The term “causing the focus of the operating system to switch from theversion of the specific virtual object presented in the extended realityenvironment to another virtual object presented in the virtual display”may be understood as implementing a task switching that transfers thefocus of the operating system from the version of the specific virtualobject external to the virtual display onto the particular virtualobject inside the virtual display. Subsequent inputs may be implementedwith respect to the particular virtual object inside the virtualdisplay.

Thus, a user may use the keyboard to switch the focus between differentvirtual objects, inside and external to the virtual display, allowingthe user to manipulate and control virtual objects anywhere in theextended reality environment. For example, a wearable extended realityappliance may present a virtual display including an editable documentand a messaging widget. The wearable extended reality appliance mayadditionally present a larger, editable version of the messaging widgetexternal to the virtual display. The user may use a keyboard to togglethe focus between the editable document inside the virtual display andthe version of the messaging widget external to the virtual display,allowing the user to switch between editing the editable document andediting a message via the version of the messaging widget.

By way of a non-limiting example, reference is now made to FIG. 16 ,which illustrates the exemplary environment of FIGS. 10 and 11 where akeyboard is provided for controlling the presentation of content inextended reality environment 1004, consistent with some embodiments ofthe present disclosure. FIG. 16 is substantially similar to FIG. 11 withthe noted difference that user 1016 is seated at desk top 1020 inposition to type via keyboard 1018. Virtual display 1002 may includegroup 1050 of multiple virtual objects, such as virtual workspace 1012and virtual house plant 1014. User 1016 may provide an input to generateversion 1014B of virtual house plant 1014 resting on desk top 1020,e.g., inside extended reality environment 1004 and external to virtualdisplay 1002. While the focus of an operating system configured withsmart glasses 1006 is on virtual workspace 1012, user 1016 may type“Alt+Tab” on keyboard 1018 to switch the focus from virtual workspace1012 (e.g., inside virtual display 1002) to version 1014B of virtualhouse plant 1014 on desk top 1020 (e.g., inside extended realityenvironment 1004 and external to virtual display 1002). Switching thefocus to version 1014B of virtual house plant 1014 may allow an inputentered by user 1016 to be applied to decrease the size of version1014B. Since version 1014B of virtual house plant 1014 is currently infocus, processing device 460 (FIG. 4 ) may decrease the size of version1014B of virtual house plant 1014 based on the input.

While the focus of the operating system is on version 1014B of virtualhouse plant 1014, user 1016 may type “Alt+Tab” on keyboard 1018 toswitch the focus from version 1014B of virtual house plant 1014 tovirtual workspace 1012 inside virtual display 1002). Switching the focusto virtual workspace 1012 may allow edits entered by user 1016 viakeyboard 1018 to be applied to virtual workspace 1012. Since virtualworkspace 1012 is now in focus, processing device 460 (FIG. 4 ) may editvirtual workspace 1012 based on inputs entered by user 1016.

Some embodiments involve, after generating the presentation of theversion of the specific virtual object in the extended realityenvironment, receiving an input from a keyboard for a thumbnail viewpresentation of virtual objects associated with the virtual display, andincluding a thumbnail version of the specific virtual object presentedin the extended reality environment in the thumbnail view presentationof the virtual objects associated with the virtual display. The term“thumbnail view presentation” may refer to a miniature, symbol, icon orsimplified depiction of a larger and/or more detailed object. A“thumbnail view presentation of virtual objects associated with thevirtual display” may include one or more miniature, symbol, icon orsimplified depictions (e.g., thumbnails) of objects affiliated with thevirtual display. A “thumbnail version of the specific virtual object”may include a miniature, symbol, icon or simplified depiction of thespecific virtual object. A group of image thumbnails may serve as anindex for organizing multiple objects, such as images, videos, andapplications. Objects affiliated (e.g., associated) with the virtualdisplay may include objects displayed inside the virtual display orderived there from. For example, a version of a virtual object displayedinside a virtual display may be associated with the virtual display,even when the version is presented external to the virtual display.Thus, after the at least one processor presents the version of thespecific virtual object external to the virtual display, the user mayrequest a thumbnail view of any objects associated with the virtualdisplay. The at least one processor may display a thumbnail view ofobjects presented inside the virtual display and any virtual objectsassociated there with. As an example, a virtual display may present aread-only virtual document. An editable version of the virtual documentmay be displayed external to the virtual display. A thumbnail viewpresentation of objects associated with the virtual display may includethe read-only virtual document as well as the editable version of thevirtual document.

By way of a non-limiting example, reference is now made to FIG. 17 ,which illustrates the exemplary system of FIGS. 10 and 11 where thecontent in the virtual display is presented as a thumbnail view,consistent with some embodiments of the present disclosure. FIG. 17 issubstantially similar FIG. 16 with the noted difference of a thumbnailview 1700 of virtual objects associated with virtual display 1002. User1016 may request to view thumbnail view 1700 via keyboard 1018, or viaother input devices (e.g., input interface 330 of input unit 202 of FIG.3 ). Upon receiving the request, processing device 460 (FIG. 4 ) may addto or replace the presentation of virtual display 1002 (FIG. 10 ),virtual objects included therein, and virtual objects derived therefrom,such as version 1014B (FIG. 16 ) of virtual house plant 1014 (FIG. 11 )with thumbnail view 1700. Thumbnail view 1700 may include a thumbnailrepresentation of each virtual object included in virtual display 1002and additionally, any virtual object associated there with, such as athumbnail representation 1702 of version 1014B of virtual house plant1014. In some implementations, processing device 460 may replace virtualdisplay 1002 and objects derived therefrom with thumbnail view 1700,e.g., as illustrated in FIG. 17 . In some implementations, processingdevice 460 may display virtual display 1002 and objects derivedtherefrom alongside thumbnail view 1700. Thumbnail representation 1702may correspond to version 1014B of virtual house plant 1014 andthumbnail representation 1704 may correspond to virtual house plant1014.

Some embodiments involve a system for extracting content from a virtualdisplay, the system including at least one processor programmed to:generate a virtual display via a wearable extended reality appliance,wherein the virtual display presents a group of virtual objects and islocated at a first virtual distance from the wearable extended realityappliance; generate an extended reality environment via the wearableextended reality appliance, wherein the extended reality environmentincludes at least one additional virtual object presented at a secondvirtual distance from the wearable extended reality appliance; receiveinput for causing a specific virtual object from the group of virtualobjects to move from the virtual display to the extended realityenvironment; and in response to receiving the input, generate apresentation of a version of the specific virtual object in the extendedreality environment at a third virtual distance from the wearableextended reality appliance, wherein the third virtual distance differsfrom the first virtual distance and the second virtual distance.

By way of a non-limiting example, FIG. 10 shows extracting content fromvirtual display 1002. The content may be extract using a system that mayinclude at least one processor (e.g., one or more of server 210 of FIG.2 , mobile communications device 206, processing device 360 of FIG. 3 ,processing device 460 of FIG. 4 , processing device 560 of FIG. 5 ). Theat least one processor may be programmed to generate virtual display1002 via smart glasses 1006 (e.g., a wearable extended realityappliance). Virtual display 1002 may present a group of virtual objects(e.g., virtual document 100, virtual widgets 1010 inside a virtual menubar 1024, a virtual workspace 1012, and a virtual house plant 1014).Virtual display 1002 may be located at a first virtual distance D1 fromsmart glasses 1006. In the implementation shown, virtual display 1002may be a flat (e.g., two-dimensional) display, and D1 may be thedistance from smart glasses 1006 to the bottom left corner of virtualdisplay 1002. The at least one processor may generate extended realityenvironment 1004 via smart glasses 1006. Extended reality environment1004 may include at least one additional virtual object, such as virtualmobile phone 1026, presented at a second virtual distance D2 from smartglasses 1006 (e.g., measured from smart glasses 1006 to the bottom leftcorner of virtual mobile phone 1026). The at least one processor mayreceive input, such as a pointing gesture by user 1016 for causing aspecific virtual object, such as virtual house plant 1014 from group1050 of virtual objects to move from virtual display 1002 to theextended reality environment 1004 (e.g., external to virtual display1002). With reference to FIG. 11 , in response to receiving the input,the at least one processor may generate a presentation of a version1014A of virtual house plant 1014 in extended reality environment 1004at a third virtual distance D3 from smart glasses 1006, where the thirdvirtual distance D3 differs from the first virtual distance D1 and thesecond virtual distance D2.

FIG. 18 illustrates a block diagram of an example process 1800 formoving content between a virtual display and an extended realityenvironment, consistent with embodiments of the present disclosure. Insome embodiments, process 1800 may be performed by at least oneprocessor (e.g., one or more of server 210 of FIG. 2 , mobilecommunications device 206, processing device 360 of FIG. 3 , processingdevice 460 of FIG. 4 , processing device 560 of FIG. 5 ) to performoperations or functions described herein. In some embodiments, someaspects of process 1800 may be implemented as software (e.g., programcodes or instructions) that are stored in a memory (e.g., any of memorydevices 212, 311, 411, or 511, or a memory of mobile device 206) or anon-transitory computer readable medium. In some embodiments, someaspects of process 1800 may be implemented as hardware (e.g., aspecific-purpose circuit). In some embodiments, process 1800 may beimplemented as a combination of software and hardware.

Referring to FIG. 18 , process 1800 may include a step 1802 ofgenerating a virtual display via a wearable extended reality appliance,wherein the virtual display presents a group of virtual objects and islocated at a first virtual distance from the wearable extended realityappliance. As described earlier, a wearable extended reality appliancemay present multiple virtual objects grouped inside a virtual displayrendered to appear as though located at particular distance from thewearer.

Process 1800 may include a step 1804 of generating an extended realityenvironment via the wearable extended reality appliance, wherein theextended reality environment includes at least one additional virtualobject presented at a second virtual distance from the wearable extendedreality appliance. As described earlier, the wearable extended realityappliance may present one or more virtual objects appearing as thoughlocated at a distance from the user different from the particulardistance to the virtual display.

Process 1800 may include a step 1806 of receiving input for causing aspecific virtual object from the group of virtual objects to move fromthe virtual display to the extended reality environment. As describedearlier, the wearable extended reality appliance may receive an input(e.g., from the user) to relocate one of the virtual objects groupedinside the virtual display, external to the virtual display.

Process 1800 may include a step 1808 of in response to receiving theinput, generating a presentation of a version of the specific virtualobject in the extended reality environment at a third virtual distancefrom the wearable extended reality appliance, wherein the third virtualdistance differs from the first virtual distance and the second virtualdistance. As described earlier, the wearable extended reality appliancemay respond to the input by presenting another rendition of the specificvirtual object external to the virtual display appearing as thoughlocated at a distance from the user (e.g., third virtual distance)different from the distance to the virtual display (e.g., the firstvirtual distance) and from the distance to the additional virtual object(e.g., the second virtual distance). The rendition of the specificvirtual object may be presented in place of, or concurrently with thespecific virtual object inside the virtual display.

The relative orientation of a wearable extended reality appliance to anassociated physical input device may correspond to an operational modefor the wearable extended reality appliance. For example, when thewearable extended reality appliance is in a first orientation relativeto the input device (e.g., the wearer is close to and facing the inputdevice), a first operational mode may be applied to interface with thewearer. Conversely, when the wearable extended reality appliance is in asecond orientation relative to the input device (e.g., the wearer isremote from, or facing away from the input device), a second (e.g.,different) operational mode may be applied to interface with the wearer.

In some embodiments, operations may be performed for selectivelyoperating a wearable extended reality appliance. A link between awearable extended reality appliance and a keyboard device may beestablished, e.g., communicatively coupling the wearable extendedreality appliance to the keyboard device. Sensor data from at least onesensor associated with the wearable extended reality appliance may bereceived. The sensor data may be reflective of a relative orientation ofthe wearable extended reality appliance with respect to the keyboarddevice. Based on the relative orientation, a specific operation mode forthe wearable extended reality appliance may be selected from a pluralityof operation modes. For example, one specific operation mode may beassociated with receiving input via the physical input device. A usercommand based on at least one signal detected by the wearable extendedreality appliance may be identified. An action responding to theidentified user command in a manner consistent with the selectedoperation mode may be executed.

In some instances, the description that follows may refer to FIGS. 19 to24 , which taken together, illustrate exemplary implementations forperforming operations for selectively operating a wearable extendedreality appliance, consistent with some disclosed embodiments. FIGS. 19to 24 are intended merely to facilitate the conceptualizing of oneexemplary implementation for performing operations for selectivelyoperating a wearable extended reality appliance and do not limit thedisclosure to any particular implementation. The description thatfollows includes references to smart glasses as an exemplaryimplementation of a wearable extended reality appliance. It is to beunderstood that these examples are merely intended to assist in gaininga conceptual understanding of disclosed embodiments, and do not limitthe disclosure to any particular implementation for a wearable extendedreality appliance. The disclosure is thus understood to relate to anyimplementation for a wearable extended reality appliance, includingimplementations different than smart glasses.

Some embodiments provide a non-transitory computer readable mediumcontaining instructions for performing operations for selectivelyoperating a wearable extended reality appliance. The term“non-transitory computer-readable medium” may be understood as describedearlier. The term “containing instructions” may refer to includingprogram code instructions stored thereon, for example to be executed bya computer processor. The instructions may be written in any type ofcomputer programming language, such as an interpretive language (e.g.,scripting languages such as HTML and JavaScript), a procedural orfunctional language (e.g., C or Pascal that may be compiled forconverting to executable code), object-oriented programming language(e.g., Java or Python), logical programming language (e.g., Prolog orAnswer Set Programming), or any other programming language. In someembodiments, the instructions may implement methods associated withmachine learning, deep learning, artificial intelligence, digital imageprocessing, optimization algorithms, and any other computer processingtechnique. The term “performing operations” may involve calculating,executing, or otherwise implementing one or more arithmetic, mathematic,logic, reasoning, or inference steps, for example by a computingprocessor. The term “wearable extended reality appliances” may refer toa head-mounted device, for example, smart glasses, smart contact lens,headsets or any other device worn by a human for purposes of presentingan extended reality to the human, as described earlier. The term“selectively operating a wearable extended reality appliance” mayinclude choosing how the wearable extended reality appliance functionsor operates, for example based on one or more criteria or conditions.Thus, program code instructions (e.g., a computer program) may beprovided (e.g., stored in a memory device of a computing device, such asany of memory devices 311 of FIG. 3, 411 of FIG. 4 , or 511 of FIG. 5 ).The program code instructions may be executable by a processing device(any of processing devices 360, 460, and 560, and mobile device 206 ofFIG. 2 ). Executing the program code instructions may cause theprocessing device to choose or elect how a wearable extended realityappliance functions (e.g., operates). For example, the wearable extendedreality appliance may function in a different manner depending on one ormore criterion, and the processing device may elect a specific manner offunctioning based on a determined criterion.

For example, the wearable extended reality appliance may be configuredto display content according to a first display configuration when thewearer is seated at a work station and display the content according toa second display configuration when the wearer is away from the workstation. As another example, the first/second display configurations maydefine a specific region of the field of view of the wearer of theextended reality appliance, a specific size, intensity, transparency,opacity, color, format, resolution, level of detail, or any otherdisplay characteristic. By way of example, when the wearer is seated atthe work station, an electronic mail application may be displayed largerand brighter than when the wearer is away from the work station. Asanother example, when the wearer is seated at the desk, the wearableextended reality appliance may be configured to present content visually(e.g., on a virtual screen), whereas when the wearer is walkingoutdoors, the wearable extended reality appliance may be configured topresent the content audibly, e.g., via a speaker.

Reference is now made to FIGS. 19 and 20 which, together, are aconceptual illustration of an environment for selectively operating awearable extended reality appliance, consistent with some disclosedembodiments. FIGS. 19 and 20 include a wearer 1900 donning a wearableextended reality appliance (e.g., a pair of smart glasses 1902). Smartglasses 1902 may be associated with a keyboard 1904 resting on a tablesurface 1912 included with a work station 1906. Smart glasses 1902 maybe configured to display content, such as a forecast weather app 1908.Processing device 460 (FIG. 4 ) may be configured to control theoperation of smart glasses 1902 based on one or more criterion. Forexample, turning to FIG. 19 , when wearer 1900 is sitting at a workstation 1906 in proximity to keyboard 1904, processing device 460 maycause smart glasses 1902 to display forecast weather app 1908 inside avirtual screen 1910 at a fixed distance from work station 1906 and/orfrom keyboard 1904 (e.g., tethered to work station 1906). Turning toFIG. 20 , when wearer 1900 is away from work station 1906, processingdevice may cause smart glasses 1902 to display forecast weather app 1908at a fixed distance from smart glasses 1902 and follow the gaze ofwearer 1900 (e.g., tethered to smart glasses 1902).

Some embodiments include establishing a link between a wearable extendedreality appliance and a keyboard device. The term “establishing” mayrefer to setting up, creating, implementing, participating in, orconstructing. The term “link” may refer to a connection that joins orcouples two separate entities, such as, a wearable extended realityappliance and a keyboard device. For example, a communications link maycouple two disparate entities to create a channel for exchanginginformation as signals. The signals may be analog (e.g., continuous) ordigital signals (e.g., discrete) and the communications link may besynchronous, asynchronous, or isochronous.

In some embodiments, the link between the wearable extended realityappliance and the keyboard is wireless. A wireless communications linkmay be established between the wearable extended reality applianceand/or the keyboard device via scanning (e.g., actively and/orpassively) and detecting nearby devices according to a predeterminedwireless communications protocol. Wireless links may be established byrecognizing authorized devices, sharing of recognized credentials, orthrough any form of pairing. Examples of wireless communicationstechnology may include transceivers for sending and receivinginformation via radio waves (e.g., Wi-Fi, Bluetooth, Zigbee, RFID, GPS,broadband, long, short or medium wave radio), microwave, mobile (e.g.,telephony) communications, infrared signals, and ultrasound signals. Insome embodiment, a wireless infrared communications link may beestablished by optically coupling an IR emitter with an IR detectorconfigured with one or both of the wearable extended reality applianceand a keyboard device. As another example, a wireless ultrasoundcommunications link may be established by coupling an ultrasound emitter(e.g., speaker) with an ultrasound receiver (e.g., microphone)configured with one or both of the wearable extended reality applianceand a keyboard device.

In some embodiments, the communications link between the wearableextended reality appliance and the keyboard is a wired connection. Forexample, a wired communications link may be created by physicallycoupling the wearable extended reality appliance and/or the keyboarddevice using a wire, cable, or fiber, for example, in compliance with awired communications standard (e.g., USB, USB-C, micro-USB, mini-B,coaxial cable, twisted cable, Ethernet cable). Other examples of wiredcommunications technology may include serial wires, cables (e.g.,multiple serial wires for carrying multiple signals in parallel such asEthernet cables), fiber optic cables, waveguides, and any other form ofwired communication technology. In some embodiments, the communicationslink may include some combination of the wired and wireless technologiesdescribed earlier.

The term “wearable extended reality appliance” may be understood asdescribed earlier. The term “keyboard device” may refer to an inputdevice including multiple keys representing alphanumeric characters(letters and numbers), and optionally, a numeric keypad, specialfunction keys, mouse cursor moving keys, and status lights. In someembodiments, the keyboard device is selected from a group consisting of:a laptop computer, a standalone network connectable keyboard, and awireless communication device having a display configured to display akeyboard. For example, the keyboard device may be a mechanical keyboard,an optical keyboard, a laser projected keyboard, a hologram keyboard, atouch sensitive keyboard (e.g., displayed on a touch-sensitiveelectronic display), a membrane keyboard, a flexible keyboard, a QWERTYkeyboard, a Dvorak keyboard, a Colemak keyboard, a chorded keyboard, awireless keyboard, a keypad, a key-based control panel, a virtualkeyboard (e.g., synthesized by a processor and displayed in an extendedreality environment), or any other array of control keys. Selecting akey of a keyboard (e.g., by pressing a mechanical key, touching a touchsensitive key, selecting a key of projected or hologram keyboard) maycause a character corresponding to the key to be stored in an inputmemory buffer of a computing device.

Thus, a communications channel (e.g., link) may be created (e.g.,established) between a wearable extended reality appliance and akeyboard device, allowing the exchange of data there between. Forexample, a user donning a wearable extended reality appliance may entera workspace including a keyboard device. The wearable extended realityappliance and the keyboard device may each include transceivers fortransmitting and receiving radio signals, for example according to aBluetooth protocol, allowing the wearable extended reality appliance andthe keyboard device to detect each other (e.g., pair) and communicatealong a Bluetooth channel.

By way of a non-limiting example, turning to FIG. 20 , network interface320 (FIG. 3 ) of keyboard device 1904 may emit a Bluetooth radio signalconfigured to be detected by a Bluetooth receiver. Network interface 420(FIG. 4 ) of smart glasses 1902 may scan for a Bluetooth radio signaland detect the Bluetooth radio signal emitted via network interface 320of keyboard device 1904. Keyboard device 1904 and smart glasses 1902 mayexchange data via network interface 320 and network interface 420,respectively, complying with a Bluetooth protocol to pair smart glasses1902 with keyboard device 1904.

Some embodiments include receiving sensor data from at least one sensorassociated with the wearable extended reality appliance, the sensor databeing reflective of a relative orientation of the wearable extendedreality appliance with respect to the keyboard device. The term“receiving” may refer to accepting delivery of, acquiring, retrieving,obtaining or otherwise gaining access to. For example, information ordata may be received in a manner that is detectable by or understandableto a processor. The processor may be local (e.g., integrated with thewearable extended reality appliance, or in the vicinity thereof, such asa local server or mobile phone) or remote (e.g., as a cloud or edgeserver). The data may be received via a communications channel, such asa wired channel (e.g., cable, fiber) and/or wireless channel (e.g.,radio, cellular, optical, IR) and subsequently stored in a memorydevice, such as a temporary buffer or longer-term storage. The data maybe received as individual packets or as a continuous stream of data. Thedata may be received synchronously, e.g., by periodically polling amemory buffer, queue or stack, or asynchronously, e.g., via an interruptevent. For example, the data may be received by any of processors 360 ofFIG. 3, 460 of FIG. 5, 560 of FIG. 6 , and/or a processor associatedwith mobile device 206 of FIG. 1 , and stored in any of storage devices311, 411, 511, or a memory of mobile device 206.

The term “sensor” may include one or more components configured todetect a signal (e.g., visible and/or IR light, radio, electric and/ormagnetic, acoustic such as sound, sonar or ultrasound, mechanical,vibration, heat, humidity, pressure, motion, gas, olfactory, or anyother type of physical signal) emitted, reflected off, and/or generatedby an object. The term “sensor data” may refer to information producedby the sensor based on a signal detected by the sensor. For example, thesensor may include a converter that converts a sensed signal to a formatconfigured for communicating to a processing device, such as anelectronic signal (e.g., for communicating via a wired communicationslink), a radio signal (e.g., for communication via a radiocommunications link), an IR signal (e.g., for communication via aninfrared communications link), or an acoustic signal (e.g., forcommunicating via an ultrasound communications link). The sensor datamay be transmitted in a binary format (e.g., as discrete bits) or ananalog format (e.g., as continuous time-variant waves).

The term “associated” may refer to the existence of an affiliation,relationship, correspondence, link or any other type of connection orcorrelation. Thus, the wearable extended reality appliance may beaffiliated (e.g., associated) with a sensor. For example, the sensor maybe mechanically coupled (e.g., physically attached) to the wearableextended reality appliance. Additionally, or alternatively, the sensormay be non-mechanically coupled (e.g., physically detached butcommunicatively coupled) to the wearable extended reality appliance,e.g., via optic, infrared, radio, ultrasound or any other type ofnon-wired communications means. In one example, the sensor may bemechanically coupled (e.g., physically attached) to the keyboard. In oneexample, the sensor may be physically separated and/or remote from boththe keyboard and the wearable extended reality appliance. In oneexample, the sensor may be associated with the wearable extended realityappliance via a data structure stored in a memory device (e.g., thememory device may be included in the wearable extended realityappliance, may be included in the keyboard, or may be external to boththe wearable extended reality appliance and the keyboard). In oneexample, the sensor may capture data associated with the wearableextended reality appliance and the association between the wearableextended reality appliance and the sensor may be through the captureddata. For example, the sensor may be an image sensor, and the captureddata may include an image of the wearable extended reality appliance.The sensor may detect signals associated with the wearable extendedreality appliance. For example, the detected signals may relate to thestate of the wearable extended reality appliance (e.g., e.g., position,orientation, velocity, acceleration, alignment, angle) relative tophysical environment of the wearable extended reality appliance. Forexample, the physical environment may include physical objects, such asa floor surface, ceiling, walls, table surface, obstructing objects(e.g., book case, house plant, person) and the detected signal mayrelate the state of the wearable extended reality appliance relative toone or more of the physical objects. The sensor may convert the detectedsignals to sensor data and transmit the sensor data to a processingdevice via a communications link as described earlier. For example, thedata may be received via an input device or sensor configured with aninput device (e.g., input unit 202 of FIG. 1 ), from mobilecommunications device (e.g., device 206), from remote processing unit(e.g., processing unit 208), by a processing device configured withsmart glasses 1902 (processing device 460 of FIG. 4 ), or from any otherlocal and/or remote source.

For example, the wearable extended reality appliance may include a GPSsensor (e.g., motion sensor 473 of FIG. 4 ) to detect a location ofwearable extended reality appliance. The GPS sensor may convert thedetected location to an electric signal that may be transmitted via awired communications link to a processing device configured inside thewearable extended reality appliance (e.g., processing device 460). Asanother example, the wearable extended reality appliance may include amotion sensor (e.g., motion sensor 472 of FIG. 4 ), such as an inertialmeasurement unit (IMU) to detect the motion (e.g., velocity andacceleration) and orientation of the wearable extended realityappliance. The IMU may convert the detected motion and orientation to aradio signal (e.g., sensor data) for transmission (e.g., via networkinterface 420) to a remote processing device (e.g., remote processingunit 208 of FIG. 2 ). As another example, a camera (e.g., image sensor)may be configured with a work station in the vicinity of the wearableextended reality appliance. The camera may be configured to capture oneor more images of the wearable extended reality appliance as the wearerapproaches the work station. The camera may convert the captured imagespixels to radio signals (e.g., sensor data), for transmitting to amobile device (e.g., mobile device 206).

The term “being reflective of” may refer to indicating, expressing,revealing, or in any other way suggesting an associated state orcondition (e.g., temporary, or steady state). For example, sensor datatransmitted by an IMU integrated with a wearable extended realityappliance may reveal (e.g., be reflective of) a motion and/or currentposition and orientation of the wearer of the wearable extended realityappliance. As another example, image data (e.g., sensor data) capturedby a camera of a wearable extended reality appliance adjacent to anobject may indicate (e.g., be reflective of) the position of thewearable extended reality appliance relative to the object.

The term “orientation” may refer to the direction (e.g., alignmentand/or position in 2D and/or 3D) in which an object (e.g., a person,animal, or thing) is pointing. The term “relative orientation” may referto an orientation (e.g., direction, alignment, distance and/or position)of an object with respect to a coordinate system or with respect toanother object. The coordinate system may be fixed (e.g., relative tothe Earth) or non-fixed (e.g., relative to a different object whoseposition and/or orientation may change with time and/or context). Thus,for example, the relative orientation between the wearable extendedreality appliance and the keyboard may be determined directly, or bycalculating the orientation of each of the wearable extended realityappliance and the keyboard relative to a fixed object (e.g., relative tothe floor, ceiling), for example based on a 3D spatial map of thephysical environment including the wearable extended reality applianceand the keyboard device (e.g., as a mesh of triangles or a fused pointcloud). The relative orientation may be determined based on anycombination of image data acquired by a camera, position, location, andorientation data acquired by an IMU and/or GPS unit configured with thewearable extended reality appliance, based on ultrasound, radio, and/orIR signals emitted and reflected off objects in the environmentincluding the wearable extended reality appliance and the keyboarddevice, data store in memory (e.g., for a stationary keyboard device),predicted behavior of the wearer of the wearable extended realityappliance, and/or any other means for tracking the relative orientationbetween the wearable extended reality appliance and the keyboard device.

Thus, a sensor may detect a relative alignment, position, distance ororientation of the wearable extended reality appliance as compared to(e.g., with respect to) the keyboard device. For example, a camerawithin imaging range of the wearable extended reality appliance and thekeyboard device may acquire an image of both the wearable extendedreality appliance and the keyboard device. The image data may beconverted to an electronic or radio signal (e.g., sensor data) andtransmitted to a processing device via a network interface. Theprocessing device may analyze the information encoded in the electronicor radio signal to determine the relative orientation of the wearableextended reality appliance and the keyboard device. The image data maythus be reflective of the orientation of the wearable extended realityappliance relative to the keyboard device. As another example, thewearable extended reality appliance may include an IR emitter, and theposition of the keyboard device may be known in advance (e.g.,physically tethered to a surface of a work station). A processing device(e.g., associated with the work station) may receive an IR signal (e.g.,sensor data) from by the IR emitter of the wearable extended realityappliance and analyze the received signal to determine the position andorientation of the wearable extended reality appliance relative to thesurface of the work station. The IR signal may thus be reflective of therelative orientation of the wearable extended reality appliance to thesurface, and thus the keyboard device tethered thereto. As anotherexample, a gyroscope of an IMU configured with the wearable extendedreality appliance may detect an orientation of the wearable extendedreality appliance. The IMU may convert the orientation to an electronicsignal (e.g., sensor data) and transmit the electronic signal to aprocessing device via a radio communications channel (e.g., Wi-Fi,Bluetooth). The processing device may analyze the received signal todetermine the orientation of the wearable extended reality appliancerelative to the keyboard device (e.g., having a known, fixed position).

By way of a non-limiting example, turning to FIG. 20 , wearer 1900donning smart glasses 1902 (such as smart glasses) may approach workstation 1906. A camera (e.g., image sensor 472 of FIG. 4 ) configuredwith (e.g., associated with) smart glasses 1902 may capture an image ofkeyboard device 1904 resting on table surface 1912 and provide the imageto processing device 460. In addition, an IMU configured with motionsensor 473 of smart glasses 1902 may sense the orientation of smartglasses 1902 relative to a floor parallel to surface 1912 and transmitthe sensed orientation to processing device 460. The image data and/ororientation data may be analyzed by processing device 460 to determinethe relative orientation of smart glasses 1902 with respect to keyboarddevice 1904, and thereby may be reflective of the relative orientationthere between.

In some embodiments, the relative orientation includes a distancebetween the wearable extended reality appliance and the keyboard device,wherein the operations further include analyzing the sensor data todetermine an indicator of the distance. The term “distance” may refer toa spatial separation or gap between the wearable extended realityappliance and the keyboard device. For example, the distance may bemeasured in absolute terms, such as a Euclidian distance measured incentimeters, meters, feet, yards, or any other distance unit (e.g.,floor tiles separating two objects accounting for only a horizontalplanar distance). For example, the distance may be measure in relativeterms, such as by measuring a distance relative to other objects (e.g.,fixed and/or mobile objects) in the environment, such as a bookcase,window, or beacon (e.g., emitting a signal). Techniques for measuringdistance between two objects may include capturing and analyzing imagesof the objects, receiving GPS signals from a GPS satellite, detectingsonar, Lidar, or radar signals emitted, reflected, and/or absorbed fromthe objects, applying interferometry (e.g., by detecting a Dopplershift) on signals emitted and/or reflected from the objects, and anyother technique for measuring distance. The term “analyzing” may referto investigating, scrutinizing and/or studying a data set, for example,to determine a correlation, association, pattern, or lack thereof withinthe data set or with respect to a different data set. The term“indicator of a distance” may refer to information allowingdetermination of the distance between two objects, for example byexpressing or revealing an effect of the distance, e.g., on a signal.For example, distance may cause attenuation, a phase or time shift on asignal, such as a light, infrared, radio, or acoustic wave. In someembodiments, image data received by an image sensor may be analyzed, forexample, using one or more image processing techniques such asconvolutions, fast Fourier transforms, edge detection, patternrecognition, and clustering to identify objects from the image pixels.Mathematical functions, such as geometric, algebraic, and/or scalingfunctions may be applied to the identified objects to determine thedistance there between. For example, a beacon may emit signals (e.g.,IR, radio, ultrasound) that may be reflected off the wearable extendedreality appliance and the keyboard device. The reflected signals may beinvestigated (e.g., analyzed), for example based on timing, angle,phase, attenuation, Doppler shift, as indicators of the distance betweenthe wearable extended reality appliance and the keyboard device.Additionally, or alternatively, the wearable extended reality appliancemay include a camera and/or a beacon emitting signals, and the distanceto the keyboard device may be determined based on images acquired by thewearable camera, and/or signals emitted by the wearable beacon reflectedoff an object (e.g., the keyboard device).

By way of a non-limiting example, turning to FIG. 19 , a camera (e.g.,image sensor 472 of FIG. 4 ) configured with smart glasses 1902 mayacquire an image of keyboard device 1904 while wearer 1900 is seated atwork station 1906. Processing device 460 may analyze the image toidentify the perspective angle and scale of keyboard device 1904, forexample, relative to other identified objects captured in the image,and/or based on absolute dimensions of keyboard device 1904 known inadvance. The image analysis may indicate the distance between smartglasses 1902 and keyboard device 1904.

In some embodiments, the relative orientation includes a facingdirection of the wearable extended reality appliance with respect to thekeyboard device, wherein the sensor data reflective of the relativeorientation includes image data, and where the operations furtherinclude analyzing the image data to determine an indicator of the facingdirection. The term “facing direction” may refer to an angle ororientation corresponding to a line-of-sight of the wearer of thewearable extended reality appliance or corresponding to at least aportion of the field of view of the wearer, e.g., at a particular pointin time. The term “image data” may refer to information acquired by acamera, e.g., as image pixels. The term “an indicator of the facingdirection” may refer to information allowing determination of theorientation of the gaze of the wearer of the wearable extended realityappliance, by expressing or revealing an effect of the gaze. Forexample, a camera configured with the wearable may be aligned with afrontal head pose of the wearer such that the camera acquires imagessubstantially corresponding to the field-of-view, or directional gaze ofthe wearer, e.g., as the wearer turns his head, the camera acquiresimages corresponding to what the wearer sees. Image data (e.g., sensordata) acquired by the camera may be provided to a processing device foranalysis. The processing device may apply image processing techniques(such as egomotion or ego-positioning algorithms) to the image data todetect the presence of the keyboard device. Based on the analysis, theprocessing device may determine the facing direction of the wearableextended reality appliance with respect to the keyboard. For example, ifthe keyboard device is positioned substantially centered in the imagedata, the processing device may determine that the facing direction ofthe wearable extended reality appliance is aligned with the keyboard.Conversely, if no keyboard is detected in the image data or the keyboardis detected in a peripheral region of the image, the processing devicemay determine that the facing direction of the wearable is not alignedwith the keyboard device, e.g., aligned with an object other than thekeyboard device. As another example, a camera tethered to the keyboarddevice may capture an image of the wearer of the wearable extendedreality appliance and the image data (e.g., sensor data) provided to aprocessing device for analysis. The processing device may apply imageprocessing techniques to the image data to detect the directional gaze(e.g., head pose) of the wearer. Based on the analysis, the processingdevice may determine the facing direction of the wearable extendedreality appliance with respect to the keyboard. In some examples, amachine learning model may be trained using training examples todetermine facing directions from images and/or videos. An exemplarytraining example may include a sample image and/or a sample video and anassociated label indicating a facing direction corresponding to thesample image and/or the sample video. The trained machine learning modelmay be used to analyze the image data and determine the indicator of thefacing direction. In some examples, at least part of the image data maybe analyzed to calculate a convolution of the at least part of the imagedata and thereby obtain a result value of the calculated convolution.Further, in response to the result value of the calculated convolutionbeing a first value, one indicator of the facing direction may bedetermined, and in response to the result value of the calculatedconvolution being a second value, a different indicator of the facingdirection may be determined.

By way of a non-limiting example, reference is now made to FIG. 21 whichis substantially similar to FIGS. 19 and 20 with the notable differencethat wearer 1900 is facing away from keyboard device 1904. Image sensor472 (FIG. 4 ) configured with smart glasses 1902 may acquire an image ofthe field of view of wearer 1900. Processing device 460 may analyze theimage data. When keyboard device 1904 is absent from or in a peripheralregion of the image data, processing device 460 may determine that thefacing direction of smart glasses 1902 is away from keyboard device1904.

In some embodiments, the sensor data reflective of the relativeorientation is received from the keyboard device while the keyboarddevice is located on a surface. The term “sensor data reflective of therelative orientation is received from the keyboard device” may beunderstood as the keyboard device reflecting and/or emitting a signalfrom which the relative orientation may be determined. For example,light reflecting off the keyboard device may be captured by a camera andconverted to image pixels (e.g., sensor data). The term “located” mayrefer to a station, placement, or position of an object. The term“surface” may include an upper layer of an object, such as a flat or topplanar layer of a supporting plank, or board; or a contoured outersurface. For example, a surface may be a topmost layer of a table, andmay be made of a hard, smooth material (e.g., wood, plastic, stone,metal, ceramic) capable of supporting other objects in a steady-state(e.g., stable) manner. For example, the keyboard device may be stationedor placed (e.g., located) on a plank or board forming an upper layer(e.g., surface) of a desk. A camera (e.g., configured with the wearableextended reality appliance and/or a work station including the surfaceand keyboard device) may sense light waves (e.g., ambient light)reflecting off the keyboard device during the time period (e.g., while)the keyboard device is positioned on the desk. The camera may convertthe sensed light waves to pixels or image data (e.g., sensor data). Aprocessing device may analyze the image data and detect the keyboarddevice and the alignment and size of the keyboard device relative toother objects detected in the image data. Based on the alignment andsize, the processing device may determine the relative orientation ofthe keyboard device and the wearable extended reality appliance.

By way of a non-limiting example, turning to FIG. 19 , image sensor 472(FIG. 4 ) configured with smart glasses 1902 may acquire an image ofkeyboard device 1904 resting on table surface 1912. Processing device460 may analyze the image to determine the distance between keyboarddevice 1904 and smart glasses 1902 (e.g., based on known absolute sizeand dimensions of keyboard device 1904), for example usingtriangulation.

Some embodiments include, based on the relative orientation, selectingfrom a plurality of operation modes a specific operation mode for thewearable extended reality appliance. The term “based on” may refer tobeing established by or founded upon, or otherwise derived from. Theterm “selecting” may refer to choosing, electing, or discriminatelypicking, for example one from multiple possible choices. For example,the selection may be performed by a processing device integrated with,local to, and/or remote from the wearable extended reality appliance orany combination thereof (such as one or more of processing devices 360of FIG. 3, 460 of FIG. 4, 560 of FIG. 5 ). For example, a selection maybe implemented by querying a database storing multiple possible choicesusing one or more criterion as filters, rules, and/or conditions for thesearch query. The database may be local or remote (e.g., with respect toa processing device implementing the search). Additionally, oralternatively, the selection may include performing one or more oflogical, inference, interpolation, extrapolation, correlation,clustering, convolution, and machine learning operations. e.g., based onone or more criterion. The criterion may be, for example, user defined,hardware defined, software defined, or any combination thereof. Thecriterion may relate, for example, to distance, orientation, alignment,communication and/or processor latency and/or bandwidth (e.g., foreither one or both of the wearable extended reality appliance andkeyboard device), use context, the type of application for which theoperation mode is being applied (e.g., work or personal use, high or lowpriority), the location of the wearer (e.g., private or public location,work or home, indoor or outdoor), the type of keyboard device (e.g.,virtual, physical, or projected), user defined preferences, systemdefined preferences, or any other criterion relating to the operation ofthe wearable extended reality appliance and/or the keyboard device.

The term “operation modes” may refer to configurations, arrangements(e.g., including one or more parameter settings, default or customsettings, preferences,) for performing or implementing one or moreactions, functions, or procedures. For example, the operation modes maybe based on one or more default settings (e.g., hardware and/orsoftware), and/or user-defined settings and preferences. In someembodiments, the operation modes may be based one use context, use type,preferences, or user needs. Examples of user needs may includevisibility and/or attention needs (e.g., based on user feedback and/ormachine learning of the behavior and/or preferences of the wearer), thepresence of noise and/or objects in the vicinity of the wearer, and anyother criterion affecting the user experience of the wearer. In someembodiments, the operation modes may be based on an efficiency goal, apower consumption goal, an emissions goal, or environmental conditions(e.g., ambient light, dust level, wind, temperature, pressure,humidity). In some embodiments, the operation modes may correspond todevice requirements of the wearable extended reality appliance and/orkeyboard device, such as processing, memory, and internal communication(e.g., bus) capacity, availability and/or limitations. In someembodiments, the operation modes may be based on communicationsrequirements of the communications network linking the wearable extendedreality appliance with the keyboard device (e.g., communicationsbandwidth capacity, availability, or latency).

For example, different operation modes may be defined for indoor versusoutdoor use of a wearable extended reality appliance. As anotherexample, different operation modes may be defined based the time of day,day of week (e.g., holiday or work day). As another example, differentoperation modes may be defined for relatively mobile uses (e.g.,regularly moving away from a work station) versus relatively stationaryuses of a wearable extended reality appliance (e.g., rarely moving awayfrom a work station). As another example, different operation modes maybe defined for when the wearer is in proximity to a work station orremote from the work station (e.g., affecting the ability to communicatewith another device tethered to the work station).

The term “specific operation mode” may refer to a distinct, special, orprecise configuration or arrangement for performing one or more action,functions, or procedures. For example, from multiple different operationmodes defined for the wearable extended reality appliance, a single(e.g., specific) operation mode may be chosen (e.g., selected) based onone or more criterion, such as any one or more of the criteriondescribed earlier.

The operation modes may be store in memory, such as a memory deviceintegrated with the wearable extended reality appliance or otherwiseaccessible by the wearable extended reality appliance. For example, theoperation modes may be stored in a memory device, such as one or more ofa memory device configured with the wearable extended reality appliance(e.g., memory device 411 of FIG. 4 ), in a memory device of a remoteprocessing unit (e.g., memory device 511 of FIG. 5 ), in a memory deviceof a mobile device (e.g., mobile device 206 of FIG. 1 ) or any othermemory device. A processing device (e.g., one or more of processingdevice 460, processing device 560, a processing device configured withmobile device 206 of FIG. 2 , or any other processing device) may accessone or more of the operation modes by querying the memory device basedon one or more rules. For example, one rule for querying the memorydevice for an operation mode may be related to the relative orientationbetween the wearable extended reality appliance and the keyboard device.

Thus, the relative orientation of the wearable extended realityappliance to the keyboard device may be used to choose a specificoperation mode from multiple candidate operation modes for wearableextended reality appliance. For example, one operation mode may besuitable for Bluetooth or Wi-Fi communication. Thus, when the wearableextended reality appliance is sufficiently close to the keyboard deviceto establish a Bluetooth or Wi-Fi communications channel, the Bluetoothor Wi-Fi operation mode may be selected, respectively. As anotherexample, an operation mode that tethers the display to the keyboarddevice may be suitable for when the wearable extended reality applianceis facing towards the keyboard device and a different operation modethat tethers the display to the directional gaze of the wearer may besuitable for when the wearer is facing away from the keyboard device. Asanother example, an operation mode presenting content audibly may besuitable for when the wearable extended reality appliance is movingquickly relative to the keyboard device (e.g., in the context of anexercise application) and a different operation mode presenting contentvisually may be suitable for when the wearer is relatively stationaryrelative to the keyboard device (e.g., in the context of editing adocument, or viewing content.

By way of a non-limiting example, turning to FIG. 19 , image sensor 472(FIG. 4 ) configured with smart glasses 1902 may capture an image ofkeyboard device 1904 and send the image pixels to processing device 460.Concurrently, motion sensor 473 configured with smart glasses 1902 maydetect that smart glasses 1902 are relatively stationary (e.g., over atime period, such as 5 seconds) and send the motion sensor data toprocessing device 460. Processing device 460 may analyze the imagepixels and the motion data received from image sensor 472 and motionsensor 473, respectively, and the relatively close and stable (e.g.,steady state) position and orientation of smart glasses 1902 withrespect to keyboard device 1904. Processing device 460 may determinethat wearer 1900 is in a sitting position at work station 1906 andfacing keyboard device 1904. In response, processing device 460 mayquery memory device 411 for an operation mode suited to sitting at workstation 1906 from multiple operation modes for smart glasses 1902 storedin memory device 411, thereby selecting a specific mode from theplurality of available operation modes for smart glasses 1902. Theselected operation mode may cause a forecast weather app 1908 to bedisplayed inside virtual screen 1910 tethered to work station 1906. Theforecast may include a high level of detail, such as the forecast forthe next twelve hours.

Turning to FIG. 20 , based on data received from image sensor 472 andmotion sensor 473, processing device 460 may determine that wearer 1900is relatively distant from keyboard device 1904 and the position andorientation of smart glasses 1902 with respect to keyboard device 1904is unstable (e.g., not in steady state). For example, a smart watch 1914tracking steps walked wearer 1900 may notify processing device 460 thatwearer 1900 is in motion (e.g., walking). In response, processing device460 may query memory device 411 for an operation mode suited to walkingwhile away from work station 1906 from the multiple operation modesstored in memory device 411, thereby selecting a specific mode from theplurality of available operation modes. The selected operation mode maycause forecast weather app 1908 to be displayed at a predefined distancefrom smart glasses 1902 (e.g., tethered to wearer 1900 while moving).The forecast may include a lower level of detail, such as the forecastonly for the next two hours.

Some embodiments include identifying a user command based on at leastone signal detected by the wearable extended reality appliance. The term“identifying” may refer to recognizing, perceiving, or otherwisedetermining or establishing an association with a known entity. Forexample, identifying may be a result of performing one or more logicaland/or arithmetic operations associated with a comparison (e.g., viaquery), inference, interpolation, extrapolation, correlation,convolution, machine learning function, and any other operationfacilitating identification. The term “user command” may refer to anorder, direction or instruction issued by an individual interfacing witha computing device. Examples of user commands may include vocalizedinstruction (e.g., detected by a microphone), head, hand, foot, and/orleg motions or gestures (e.g., detected by a camera and/or an IMU or GPSdevice), eye motions (e.g., detected via an eye tracker), data entered,or selections made via a manual input device (e.g., buttons, switches,keyboard, electronic pointing device, touch-sensitive screen), dataentered or selections made via a foot-operated device (e.g., pedal,footswitch), and/or any other technique for interfacing between a userand a computing device. The term “identifying a user command” mayinvolve recognizing the user command, e.g., based on a comparison,correlation, clustering algorithm, or any other identificationtechnique. For example, a vocalized user command may be identified byinvoking voice recognition software. As another example, a head, eye,hand, foot, and/or leg gesture or motion command may be identified viagesture and/or motion recognition software. As another example, acommand entered as data via in input interface of a computing device maybe identified by an event listener configured with an operating systemrunning on the computing device. Examples of user commands may include arequest to invoke, close, or change the execution of an application on adevice, turn on/off a device or device setting and/or change theoperation of a device, send and/or receive a notification or document,retrieve, upload, store, or delete a notification or document, orperform any other user-invoked activity.

The term “signal” may refer to an information transmission. A signal,for example, may involve a function that can vary over space and time toconvey information observed about a phenomenon via a physical medium.For example, a signal may be implemented in any range of theelectromagnetic spectrum (e.g., radio, IR, optic), as an acoustic signal(e.g., audio, sonar, ultrasound), a mechanical signal (e.g., motion orpressure on a button or keyboard), as an electric or magnetic signal, orvia any other physical medium. For example, the phenomenon may relate toa state, presence or absence of an object, an occurrence or developmentof an event or action, or lack thereof. For example, light waves (e.g.,signals) reflecting off a body in motion and/or performing a gesture maybe detected by a camera and stored as image data. As another example,motion and/or a gesture may be detected by a motion detector (e.g., IMU,GPS signals).

The term “detected” may refer to sensing (e.g., discovering ordiscerning) information embedded or encoded in a signal via a sensor(e.g., detector) corresponding to the signal type. Examples of detectorscorresponding to signal types may include an antenna detectingelectro-magnetic signals, a camera detecting optical and/or infraredsignals, a microphone detecting acoustic signals, electrical and/ormagnetic sensors detecting electric and/or magnetic fields (e.g., inanalog electronic circuitry), semiconductor diodes or switches detectingan electric current or voltage (e.g., consequent to the performing ofone or more logical operations, such as based on a user input), and anyother type of detector capable of sensing a signal.

For example, the wearable extended reality appliance may include adetector integrated thereon. For example, the detector may be an audiosensor (e.g., audio sensor 472 of FIG. 4 ), an image sensor (e.g., imagesensor 472), a motion sensor (e.g., motion sensor 472), an environmentalsensor (e.g., environmental sensor 474), and additional sensors (e.g.,sensors 472). The detector may sense an incoming analog signal (e.g.,sound, light, motion, temperature) and convert the analog signal to ananalog electronic signal via a transducer. The analog electronic signalmay be processed (e.g., using a filter, transform, convolution,compression) and converted to a digital format (e.g., encoded as bits)via an analog-to-digital converter. The digitized signal may be storedin memory (e.g., memory device 411). A processing device (e.g.,processing device 460) may retrieve the digitized signal and apply oneor more digital signal processing techniques, such as a digital filter,a smoothing algorithm, a transformation, a convolution, a correlation, aclustering algorithm, or any other digital signal processing technique.The processing device may compare the processed signal to a database ofuser commands (e.g., store in memory device 411). Based on thecomparison, the processing device may identify the user command, forexample if the digitized signal matches a predefined user command withina threshold, such as a cluster associated with the user command.Optionally, the processing device may apply a machine learning algorithmto identify the digitized signal as a user command.

In some embodiments, the user command includes a voice command and theat least one signal is received from a microphone included in thewearable extended reality appliance. The term “voice command” may referto a user command implemented by speaking words or uttering predefinedsounds associated with a user command. The term “microphone” may referto an audio sensor or voice input device as described earlier. Themicrophone may generate an audio signal that may be digitized and storedin a memory device. A processing device may apply a voice recognitionalgorithm to the digitized audio signal to identify the user command.

By way of a non-limiting example, turning to FIG. 19 , audio sensor 471(FIG. 4 ) configured with smart glasses 1902 may detect a sound producedby a words uttered by wearer 1900. Audio sensor 471 may generate anaudio signal corresponding to the detected sound. The audio signal maybe sampled (e.g., digitized) and stored in memory device 411 (e.g., asbits). Processing device 460 may apply a speech recognition algorithm tothe digitized sound to identify a sequence of words associated with auser command.

In some embodiments, the user command includes a gesture and the atleast one signal is received from an image sensor included in thewearable extended reality appliance. The term “gesture” may refer to amovement or sequence of movements of part of the body, such as a hand,arm, head, foot, or leg to express an idea or meaning. A gesture may bea form of non-verbal or non-vocal communication in which visible bodilyactions or movements communicate particular messages. A gesture may beused to communicate in place of, or in conjunction with vocalcommunication. For example, raising a hand with the palm forward may bea hand gesture indicating to stop or halt an activity, and raising athumb with the fist closed may be a hand gesture indicating approval. Acamera (e.g., image sensor) associated with the wearable extendedreality appliance may capture one or more images of a gesture performedby the wearer (e.g., using the hand, arm, head, foot, and/or leg). Thecamera may store the image pixels in a memory device. A processingdevice may analyze the image pixels using a gesture recognitionalgorithm to identify the gesture as the user command.

By way of a non-limiting example, turning to FIG. 22 , image sensor 472(FIG. 4 ) configured with smart glasses 1902 may detect light reflectedoff a hand of wearer 1900 and convert the reflected light as imagepixels. Processing device 460 may apply a gesture recognition algorithmto the image pixels to identify a hand gesture (e.g., pointing of theindex finger) associated with a user command. For example, the usercommand may be associated with invoking forecast weather app 1908.

As another example, a motion detector (e.g., IMU) configured with thewearable extended reality appliance may detect a head gesture performedby the wearer and convert the head gesture to an electronic signal via atransducer. The electronic motion signal may be digitized (e.g.,sampled) and stored in a memory device. A processing device may apply ahead gesture recognition algorithm to the digitized motion signal toidentify the user command.

In some embodiments, identifying the user command may account forcontext and/or circumstances associated and/or unassociated with a userissuing the user command. For example, identifying the user command maytake into account the time of day, and/or physical environment, location(e.g., public or private), a history of the user issuing the usercommand (e.g., habits and behavior based on machine learning), a contextof the user command (e.g., based on actions performed immediately priorto the user command), and any other criterion relevant to identifying auser command.

Additionally, or alternatively, a user command may be identified basedon a signal detected by the wearable extended reality appliance byreceiving the signal from an additional device (e.g., smart watch,mobile phone) in communication (e.g., wireless communication) with thewearable extended reality appliance. For example, the wearer of thewearable extended reality appliance may enter text into an applicationof a mobile phone and the mobile phone may send a notification to aprocessing device configured with the wearable extended realityappliance (e.g., processing device 460 of FIG. 4 ). As another example,the wearer of the wearable extended reality appliance may push a buttonon a smart watch and the smart watch may send a notification to aprocessing device configured with the wearable extended realityappliance (e.g., processing device 460 of FIG. 4 ).

By way of a non-limiting example, turning to FIG. 20 , a microphone(e.g., audio sensor 471 of FIG. 4 ) configured with smart glasses 1902may sense a sound emitted by wearer 1900. The microphone may convert thesound to an electronic signal, which may be digitized (e.g., viasampling) and store in memory device 411. Processing device 460 mayretrieve the digitized sound from memory device 411 and perform a voicerecognition algorithm to identify the words “Open Weather Forecast”,corresponding to a user command for invoking forecast weather app 1908,thereby detecting the user command. As another example, wearer 1900 maydon a smart watch 1914 communicatively coupled to smart glasses 1902.Wearer 1900 may press a button of smart watch 1914 associated withinvoking forecast weather app 1908. Smart watch 1914 may transmit anotification to smart glasses 1902 indicating the button press (e.g.,via a Bluetooth link). Processing device 460 may receive thenotification (e.g., detect the signal) and determine an association witha user command to invoke weather application 108, thereby detecting theuser command.

Some embodiments include executing an action responding to theidentified user command in a manner consistent with the selectedoperation mode. The term “executing” may refer to carrying out orimplementing one or more operative steps. For example, a processingdevice may execute program code instructions to achieve a targeted(e.g., deterministic) outcome or goal, e.g., in response to receivingone or more inputs. The term “action’ may refer to the performance of anactivity or task. For example, performing an action may includeexecuting at least one program code instruction (e.g., as describedearlier) to implement a function or procedure. The action may beuser-defined, device or system-defined (e.g., software and/or hardware),or any combination thereof. The action may correspond to a userexperience (e.g., preferences, such as based on context, location,environmental conditions, use type, user type), user requirements(attention or visibility limitations, urgency or priority of the purposebehind the action), device requirements (e.g., computationcapacity/limitations/latency, resolution capacity/limitations, displaysize capacity/limitations, memory and/or bus capacity/limitations),communication network requirements (e.g., bandwidth, latency), and anyother criterion for determining the execution of an action, e.g., by aprocessing device. The action may be executed by a processing deviceconfigured with the wearable extended reality appliance, a differentlocal processing device (e.g., configured with a device in proximity tothe wearable extended reality appliance), and/or by a remote processingdevice (e.g., configured with a cloud server), or any combinationthereof.

Thus, “executing an action responding to the identified command” mayinclude performing or implementing one or more operations in reaction toan identified user command, e.g., to address the user command or inassociation with or correspond to the user command. For example, uponreceiving a request from a user for data, a computing device may query adatabase stored on a memory device to locate and retrieve the requesteddata, thereby executing an action responding to the identified usercommand. As another example, upon receiving a voice command from a userto send a message to a second user, a computing device may apply a voicerecognition algorithm to identify the user command, query a tablestoring a device ID associated with the second user, establish acommunications link with the computing device of the second user (e.g.,based on the device ID), and transmit the message over thecommunications link, thereby executing an action responding to theidentified user command.

The term “in a manner consistent with” may refer to complying with oneor more predefined rules or conditions, meeting one or morerequirements, or keeping within defined limitations, settings, orparameters, e.g., defined for the selected operation mode. For example,compliance with a selected operation mode may relate to specifyingdisplay parameters, such as the resolution, color, opacity, size andamount of rendered content. As another example, compliance with aselected operation mode may relate to available memory, processor and/orcommunications bandwidth, latency requirements (e.g., to display less orlower resolution content under lower bandwidth capacity and more orhigher resolution content under higher bandwidth capacity). As anotherexample, compliance with a selected operation mode may relate to userdefined preferences (e.g., to display less detail while the user iswalking and more detail when the user is stationary). By way of anotherexample, compliance with a selected operation mode may relate toenvironmental conditions (e.g., to replace a visual display with anaudible representation under bright sunlight or very windy conditions,or conversely to replace an audible rendition with a visual displayunder noise conditions). As another example, compliance with a selectedoperation mode may relate to a location of the wearer of the wearableextended reality appliance (e.g., content may be rendered differently atwork versus at home). As another example, compliance with a selectedoperation mode may relate to the current activity (e.g., sitting,walking, driving, lying down) of the wearer, e.g., the display ofcontent may be limited while the wearer is driving.

Thus, an action performed in response to a user command may be performedin compliance with the operation mode of the wearable extended realityappliance (e.g., based on the relative orientation to the keyboarddevice). For example, a sensor may detect a wearable extended realityappliance while in motion and located beyond a threshold distance from akeyboard device. Thus, the orientation between the wearable extendedreality appliance and the keyboard device may change dynamically due tothe wearer walking. This may correspond to a walking mode for theextended reality appliance when the wearer is away from the keyboarddevice. The walking mode may include settings to present content in amanner to avoid distracting the wearer while walking (e.g., to avoidhaving to read text). While walking, the wearer of the wearable extendedreality appliance may vocalize a command to receive a message. Inresponse, the message may be presented audibly via a speaker using aspeech synthesizer, e.g., consistent with the walking mode.

As another example, a sensor may detect a wearable extended realityappliance in a second orientation (e.g., leaning back in a chair inproximity to but facing away from the keyboard device). The secondorientation may correspond to a resting mode for the wearable extendedreality appliance. The resting mode may include settings to presentcontent visually via the wearable tethered to the line-of-sight of thewearer, e.g., to allow the wearer to view content while facing away fromthe keyboard device. In response to the user command, the message may bepresented visually locked to the line-of-sight of the wearer, e.g.,consistent with the resting mode allowing the wearer to read the messagewhile facing away from the keyboard device.

As another example, a sensor may detect a wearable extended realityappliance in a third orientation (e.g., sitting upright in a chairadjacent to the keyboard device). The third orientation may correspondto a work mode for the wearable extended reality appliance. The workmode may include settings to present content visually via the wearabletethered to the work station, e.g., the keyboard device, allowing thewearer to view content displayed in a virtual screen above the keyboarddevice. In response to the user command, the message may be presentedvisually in the virtual screen, tethered to the work station andkeyboard device.

By way of a non-limiting example, turning to FIG. 19 , a camera (e.g.,image sensor 472 of FIG. 4 ) may capture an image of keyboard device1904 in proximity to wearer 1900 and at an angle indicating that weareris seated upright facing keyboard device 1904. Processing device 460 mayanalyze the image and determine a close-upright orientation betweensmart glasses 1902 and keyboard device 1904. Based on the close-uprightorientation, a close-upright mode may be selected for smart glasses1902. For example, the close-upright mode may be associated withdisplaying more content than when positioned far from keyboard device1904 and displaying content inside virtual screen 1910 locked (e.g.,tethered) to work station 1906. Wearer 1900 may vocalize the usercommand “Open weather forecast”, corresponding to invoking a weatherapplication and displaying the weather forecast. Processing device 460may display forecast weather app 1908 to include the forecast for thenext twelve hours. Forecast weather app 1908 may be displayed insidevirtual screen 1910, locked to work station 1906 and keyboard device1904, e.g., in a manner consisted with the close-upright mode.

By way of another non-limiting example, turning to FIG. 20 , a camera(e.g., image sensor 472 of FIG. 4 ) may capture an image of keyboarddevice 1904 beyond a predefined threshold from wearer 1900. Processingdevice 460 may analyze the image and determine a remote orientationbetween smart glasses 1902 and keyboard device 1904. Based on the remoteorientation, a remote operation mode may be selected for smart glasses1902. For example, the remote operation mode may be associated withdisplaying less content than when positioned in proximity to keyboarddevice 1904, and displaying content locked (e.g., tethered) to theline-of-sight of wearer 1900. Wearer 1900 may vocalize the words “Openweather forecast”, corresponding to a user command to invoke a weatherapplication and display the weather forecast. Processing device 460 maydisplay the weather forecast for the next two hours and locked to theline-of-sight of wearer 1900, e.g., in a manner consisted with theremote operation mode.

Some embodiments include accessing a group of rules associating actionsresponding to user commands with relative orientations between thekeyboard device and the wearable extended reality appliance, determiningthat the relative orientation corresponds to a specific rule of thegroup of rules, and implementing the specific rule to execute anassociated action responding to the identified user command. The term“accessing” may refer to obtaining. e.g., at least for the purpose ofreading, or acquiring relevant information. For example, data may beaccessed by a processing device querying a data store, such as adatabase. The term “group of rules” may refer to a set of guidelines,regulations, or directives. A group of rules may include general rules,or may include rules defined for a specific device, user, system, time,and/or context. Thus, a group of rules may be stored in a database on amemory device (e.g., local and/or remote) and accessed via query. Thememory device may be accessible only for privileged users (e.g., basedon a device and/or user ID) or generally accessible. The term“associating” may refer to linking, tying, relating, or affiliating.

Thus, the group of rules may link or relate one or more actions to oneor more relative orientations between the keyboard device and thewearable extended reality appliance. The actions (e.g., linked torelative orientations via the rules) may be performed in response touser commands. For example, an action to invoke an application may beperformed in response to a vocalized user command “invoke app”.Moreover, one or more rules may define how to perform the action, inother words, how to invoke the application. In particular, a rule maydefine how to invoke the application based on the relative orientationbetween the wearable extended reality appliance and the keyboard. Forexample, when the wearable extended reality appliance is withinBluetooth communication range of the keyboard (e.g., a firstorientation), a rule may cause a first version of the application to beinvoked. However, when the wearable is outside Bluetooth communicationrange from the keyboard (e.g., a second orientation), a rule (e.g., thesame or different rule) may cause a second version of the application tobe invoked. For example, the first version of the application mayinclude more content and/or may be displayed in a larger format than thesecond version.

As another example, an action to display content may be performed inresponse to a hand gesture (e.g., user command) performed by a wearer ofa wearable extended reality appliance. However, a rule may define how todisplay the content, depending on the orientation between the wearableextended reality appliance and the keyboard device. For example, whenthe wearable extended reality appliance is facing towards the keyboard(e.g., a third orientation), a rule may cause content to be displayedaccording to a first color scheme, and when the wearable extendedreality appliance is facing away from the keyboard (e.g., a fourthorientation), the same or different rule may cause content to bedisplayed according to a second color scheme. For example, the firstcolor scheme may be suitable for displaying content against a blank wallpositioned behind the keyboard device, and the second color scheme maybe suitable for displaying content suspended (e.g., floating) in a roomtogether with other distracting objects.

The term “determining” may refer to performing a computation, orcalculation to arrive at a conclusive or decisive outcome. The term“specific rule of the group of rules” may refer to a distinct, special,or precise rule from multiple different rules. Thus, a particular (e.g.,specific) rule may be selected from a group of rules based on acomputation resulting in a decisive conclusion. For example, when therelative orientation changes dynamically the processing device maycalculate an associated velocity and query for a specific rulecorresponding to the velocity. As another example, when the relativeorientation indicates a stationary position facing a blank wall, theprocessing device may query for a specific rule corresponding to viewingcontent on a blank wall while in a stationary position.

The term “corresponds” may refer to correlated with, or in conformancewith. Thus, a relative orientation (e.g., between a user and a device,or between two devices) may be used to decide (e.g., determine) whichrule to apply when performing an action. For example, a first rulecausing content to be presented using a small format may correspond to afirst relative orientation (e.g., when the relative distance between awearable extended reality appliance and a keyboard device is large,e.g., above a threshold). A second rule causing content to be presentedusing a large format may correspond to a second relative orientation(e.g., when the relative distance between the wearable appliance and thekeyboard device is small, e.g., below a threshold). Thus, in response toa user command to display content, if the wearable extended realityapparatus is beyond the threshold from the keyboard device (e.g., thefirst relative orientation), the first rule may be applied to displaycontent (e.g., using the small format). When the wearable extendedreality apparatus is within the threshold of the keyboard (e.g., thesecond relative orientation), the second rule may be applied to displaycontent (e.g., using the larger format).

The term “implementing” may refer to materializing, fulfilling, carryingout, or applying. For example, implementing a rule may cause the rule tobe applied when performing an action. The term “associated actionresponding to a user command” may refer to an action (e.g., performed inresponse to a user command, as defined earlier) that is linked, orcorresponds to a relative orientation between the wearable extendedreality appliance and the keyboard device by the rule. Thus, the rulemay create the association (e.g., link) between the action performed inresponse to a user command and the relative orientation between thewearable extended reality appliance and the keyboard device. Forexample, changing the relative orientation of the wearable extendedreality appliance may affect the ambient lighting conditions, and/orintroduce virtual and/or real obstructions. The changed orientation maythus affect how displayed content appears to the wearer of the wearableextended reality appliance (e.g., the result of executing an action inresponse to a user command). The specific rule may address the effect ofthe changed orientation by modifying one or more display settings (e.g.,to accommodate the different ambient lighting, or obstruction). Forexample, the specific rule may rearrange content and adjust thebrightness to produce a more satisfactory user experience.

For example, a user command “display message” may cause a wearableextended reality appliance to present a message including an image(e.g., perform an action responding to an identified user command).However, the relative orientation of the wearable to the keyboard devicemay determine how the message and image will be presented, based on oneor more predefined rules. For example, when the wearable extendedreality appliance is in motion and moving away from the keyboard device(e.g., the wearer is walking away from the keyboard) a first rule may beimplemented to present the message and accompanying image audibly, e.g.,to avoid distracting the wearer with visual content while walking and toconserve communications bandwidth as the wearer exits Bluetooth range ofthe work station. When the wearable extended reality appliance is inmotion and moving towards from the keyboard device, (e.g., the wearer iswalking towards the keyboard), a second rule may be implemented topresent the message audibly but present the accompanying image visually,e.g., to avoid distracting the wearer with text while walking, but allowthe wearer to see the image since there is sufficient bandwidth. Whenthe wearable extended reality appliance is stationary and facing thekeyboard device, (e.g., the wearer is sitting at the work station facingthe keyboard), a third rule may be implemented to present the messageand accompanying image visually, e.g., to provide a full visualexperience to the wearer and avoid distracting while working.

By way of a non-limiting example, turning to FIGS. 19 and 20 , inresponse to the voice command “Open weather forecast” (e.g., a usercommand), smart glasses 1902 may be configured to display the forecast(e.g., perform an action responding to a user commands). However, basedon the relative orientation between smart glasses 1902 and keyboarddevice 1904, one or more rules may be defined governing how the actionis to be performed. Referring to FIG. 19 , wearer 1900 is shown sittingat work station 1906. Processing device 460 (FIG. 4 ) may determine therelative orientation between smart glasses 1902 and keyboard device 1904(e.g., in proximity, facing, and stationary). Processing device 460 mayquery a rules database for a rule to apply when smart glasses 1902 arein proximity to, facing, and stationary with respect to keyboard device1904 (e.g., determine that the relative orientation corresponds to aspecific rule). The selected rule may define parameters for renderingcontent suitable to situations where a wearer of a pair of smart glassesis sitting at a work station, facing a keyboard. For example, the rulemay cause content to be displayed in a virtual screen tethered to thework station, using a large format and rich with details. Accordingly,processing device 460 may implement the rule to present forecast weatherapp 1908, using a large display format and including the weather for thenext twelve hours, inside virtual screen 1910, (e.g., implement thespecific rule to execute an associated action responding to theidentified user command).

Referring to FIG. 20 , wearer 1900 is shown walking towards work station1906. Processing device 460 may determine the relative orientationbetween smart glasses 1902 and keyboard device 1904 (e.g., distant,facing, and moving towards). Processing device 460 may query the rulesdatabase for a rule to apply when smart glasses 1902 are distant from,facing, and moving towards keyboard device 1904. The rule may causecontent to be displayed in a virtual screen tethered to smart glasses1902, using a small format with few details. Accordingly, processingdevice 460 may implement the rule to present forecast weather app 1908,using a small display format and including the weather for only the nexttwo hours, locked to the directional gaze of smart glasses 1902.

In some embodiments, the relative orientation includes both distanceinformation and facing information, and wherein the operations furtherinclude, for distances within a threshold: when the wearable extendedreality appliance is facing the keyboard device, selecting a firstoperation mode, and when the wearable extended reality appliance isfacing away from the keyboard device, selecting a second operation mode.The term “distance information” may include one or more measurements,assessments, or evaluations indicating an amount of space (e.g.,distance as described earlier) separating two objects, such as between awearable extended reality appliance and a keyboard device.

The term “facing information” may include one or more measurements,assessments, or evaluations indicating a facing direction as describedearlier. For example, the facing information may include a relativeangle there between, a vertical (e.g., height) disparity, a planar(e.g., horizontal) disparity, an orientation relative to another objectin the vicinity (e.g., a wall, floor, bookcase), and/or with respect tothe Earth (e.g., based on a compass). For example, a processing devicemay apply a facial recognition algorithm to an image acquired from theperspective of the keyboard device to determine if the wearer is facingthe keyboard device. Additionally, or alternatively, a processing devicemay apply an object recognition algorithm to an image acquired from theperspective of the wearable extended reality appliance to detect thepresence of the keyboard device in the field of view of the wearer, todetermine the facing information.

The term “distances within a threshold” may refer to a position insideor within a zone, for example demarcated by a boundary, limit, or border(e.g., threshold) marking the zone. For example, a threshold may bedefined relative to another object (e.g., fixed or mobile), based on theability of a device to interface with another device, based on a sensorycapability inside the zone (e.g., within the threshold), and/or lack ofsensory capability outside the zone (e.g., beyond the threshold). Forexample, when the distance between a wearer of a wearable appliance anda virtual screen is within reading range, the distance may be within thethreshold. As another example, when a device is within Bluetooth rangeof another device, the distance between the two devices may be withinthe threshold. As another example, a threshold may demarcate a zonebased on a focus or attention capability of a user (e.g., based onenvironmental conditions such as ambient light, ambient noise, or thepresence of distracting objects, or people). For example, the distancebetween a wearer of a wearable appliance and another device may bedetermined to be within the threshold based on one or more environmentalconditions. As another example, a threshold may demarcate a zone basedon user preferences or user behavior, e.g., determined based on userinput and/or machine learning. For example, when the distance between awearer of a wearable appliance and a keyboard device is within a manualtyping range, the distance may be within the threshold. A processingdevice (e.g., processing device 460 of FIG. 4 ) may determine when awearable extended reality appliance is within a threshold of a keyboarddevice based on the distance information received with the relativeorientation.

The term “facing the keyboard device” may refer to a pose of thewearable extended reality appliance substantially aligned with orpointing in a direction toward the keyboard device, such that a wearerof the wearable extended reality appliance sees the keyboard device in asubstantially centered region of his field of view. The term “facingaway from the keyboard device” may refer to a pose of the wearableextended reality appliance substantially unaligned with or pointing in adirection away from the keyboard device, such that a wearer of thewearable extended reality appliance does not see the keyboard device orsees the keyboard device in a peripheral region of his field of view.Thus, the relative orientation may include data relating to the distancebetween the wearable and the keyboard (e.g., distance information), andadditionally, data relating to the directional gaze of the wearer of thewearable with respect to the keyboard device (e.g., facing information).Either one or both of the distance information and the facinginformation may be used to determine the operation mode for the wearableextended reality appliance.

For example, when the wearable extended reality appliance issufficiently close to the keyboard device (e.g., to establish aBluetooth connection), a Bluetooth operation mode may be applied to thewearable extended reality appliance. For example, the Bluetooth channelmay allow displaying content using a high resolution. However, thedirectional gaze of the wearer (e.g., the facing information) may beused to determine additional settings for the wearable extended realityappliance. For example, when the wearable is facing the keyboard device,an operation mode corresponding to a forward-facing pose may be appliedto display content on a virtual screen tethered to the keyboard device,and when the wearable extended reality appliance is facing to the side(e.g., away from the keyboard device), an operation mode correspondingto the side-facing pose may be applied to display content on a virtualscreen tethered to the wearable extended reality appliance.

By way of a non-limiting example, turning to FIGS. 19 and 21 ,processing device 460 (FIG. 4 ) may detect smart glasses 1902 positionedwithin 50 cm of keyboard device 1904 (e.g., distance information). Forexample, processing device may receive image data from image sensor 472and analyze the image data to determine the distance information. Basedon the distance information, processing device 460 may determine thatthe distance between smart glasses 1902 and keyboard device falls withina threshold for presenting content visually with a high level of detail.For example, a weather forecast may be displayed visually and includethe forecast for the next twelve hours. Additionally, processing device460 may determine where to display the weather forecast based on thedirectional gaze of wearer 1900. For example, processing device 460 mayreceive facing information from an IMU (e.g., motion sensor 473)configured with smart glasses 1902 to determine the directional gaze.Turning to FIG. 19 , based on the facing information, processing device460 may detect a forward-facing pose for smart glasses 1902 and select aforward-facing operation mode (e.g., from multiple operation modesstored in memory device 411). For example, the forward-facing operationmode may cause twelve-hour forecast 1908 to be displayed on virtualscreen 1910 tethered (e.g., docked) to work station 1906. Turning toFIG. 21 , based on the facing information, processing device 460 maydetect a side-facing pose for smart glasses 1902 and select aside-facing operation mode for smart glasses 1902. For example, theside-facing operation mode may cause twelve-hour forecast 1908 to bedisplayed tethered to smart glasses 1902.

Some embodiments include determining that the user command is a query,and wherein in the first operation mode, the action responding to thequery involves providing a first response that includes displayinginformation on a virtual display screen associated with the keyboarddevice, and in the second operation mode, the action responding to thequery involves providing a second response that excludes from displayinginformation on the virtual display screen. The term “query” may refer toa search request, question, or inquiry. A query may include conditionsthat must be fulfilled, such as filters or rules that narrow down thesearch results. For example, a query may be formulated using a querylanguage (e.g., SQL, OQL, SPARQL, OntoQL, and any other query language),and a processing device may determine the user command is a query byidentifying query language terms in the user command. As anotherexample, a query may target a database or knowledge base, and theprocessing device may determine the user command is a query based on thetarget. A processing device may receive a user command and determinethat the user command is a query. For example, if the user command istext (e.g., entered via the keyboard device), a processing device mayparse and tokenize the text to identify the query. As another example,if the user command is a voice command, a processing device may apply aspeech recognition package to the voice command to tokenize and identifyspoken words as the query. By way of another example, if the usercommand is a gesture, a processing device may apply a gesturerecognition package to identify the query.

The term “action responding to the query” may be understood as an actionreacting or replying to a user command, as described earlier, where theuser command is a query. The term “response” may include informationreceived in reply or reaction to submitting a query. For example, aresponse to a query for an address for a contact name may be a singleaddress for a single contact with the name), several addresses for thecontact), several contacts with the same name all having one address,each with one or more addresses).

The term “displaying” may refer to selectively activating pixels of anelectronic display device or otherwise visually presenting information(e.g., a viewer of a wearable extended reality appliance). The term“displaying information” may refer to selectively activating pixels orotherwise presenting information to convey a facts and/or knowledge. Forexample, a processing device of the wearable extended reality appliancemay display content on a virtual screen by selectively activatingcertain pixels of the viewer to render virtual content overlaid on thephysical environment viewable via transparent portions of the viewer.The term “excludes” may refer to omitting or precluding.

Thus, if the wearable extended reality appliance is facing the keyboarddevice (e.g., the first operation mode is selected), a response to thequery may display information on a virtual display associated with thekeyboard device. If the wearable extended reality appliance is facingaway from the keyboard device (e.g., the second operation mode isselected), a response to the query may omit (e.g., exclude) informationfrom being display on the virtual display associated with the keyboarddevice. For example, the information may be presented audibly, in atactile manner, or on a virtual screen associated with the wearableextended reality appliance. As another example, the query may be arequest for directions to a target location. When the wearer is facingthe keyboard device, the directions may be displayed on a virtual screenabove the keyboard device (e.g., in the line-of-sight of the wearer).When the wearer is facing away from the keyboard device, however, thedirections may be recited audibly via a speaker, and/or displayed in avirtual screen tethered to the wearable extended reality display.

By way of a non-limiting example, turning to FIGS. 19 and 21 , wearer1900 submits a query by vocalizing the words “open weather forecast”.Processing device 460 (FIG. 4 ) may submit a query to a remote weatherserver (not shown) via network 214 (FIG. 2 ), for example using aweather APT, and retrieve the current weather forecast. Turning to FIG.19 , based on the front-facing orientation of smart glasses 1902 withrespect to keyboard device 1904, processing device 460 may apply thefirst operation mode and display forecast weather app 1908 in virtualscreen 1910, tethered to work station 1906 and keyboard device 1904.Turning to FIG. 21 , based on the side-facing orientation of smartglasses 1902, e.g., away from keyboard device 1904, processing device460 may apply the first operation mode and exclude forecast weather app1908 from being displayed in virtual screen 1910. Instead, processingdevice 460 may display forecast weather app 1908 tethered to smartglasses 1902.

In some embodiments, the first response includes visually presenting amost likely answer to the query and at least one alternative answer tothe query; and the second response includes audibly presenting the mostlikely answer to the query without presenting the at least onealternative answer to the query. The term “visually presenting” mayinclude displaying information on a virtual and/or physical display. Theterm “audibly presenting” may include conveying information by playingan audio signal via a speaker and voice synthesizer. The term “a mostlikely answer” may refer to a response having a high degree of certaintyassociated with the response. Thus, for example, when there are multiplepossible responses to a query, the responses may be assigned aprobability. In one example, the response may be based on an output of amachine learning model trained using training examples to generateanswers to queries. An example of such a training example may include asample query, together with a desired answer to the sample query. In oneexample, the machine learning model may be a generative model, or morespecifically text generation model. In one example, the machine learningmodel may provide two or more possible answers, and may provideconfidence level associated with each possible answer. Each possibleresponse may be based on a possible answer, and the probability assignedto the possible response may be a mathematical function (such as alinear function, a nonlinear function, a polynomial function, etc.) ofthe confidence level. And a response having a maximum probability may beselected from among the possible responses as the “most likely answer.”The term “alternative answer” may refer to a response to a querydifferent than a probably or expected response to a query, e.g., havinga relatively lower probability. Thus, when the wearer is facing thekeyboard device (e.g., the first operation mode is applied to displaythe first response on the virtual screen associated with the keyboarddevice), the most probable answer to the query may be displayed togetherwith another, e.g., alternative answer. For example, the query mayrequest a driving route to a destination, and in response, the shortestdriving route may be visually presented (e.g., on a map) together withan additional (e.g., longer) route. When the wearer is facing away fromthe keyboard device (e.g., the second response is provided that avoidsdisplaying information on the virtual screen associated with thekeyboard device), the most probable response may be vocalized via aspeaker, without providing the alternative response. For example, inresponse to the query requesting the driving route, the shortest drivingroute may be described audibly via a speaker, and audible description ofthe additional route may be omitted.

By way of a non-limiting example, turning to FIGS. 19 and 23 , wearer1900 submits a query to receive a weather forecast by issuing a voicecommand “Open weather forecast”. FIG. 23 is substantially similar toFIG. 19 with the notable different that wearer 1900 is facing away fromkeyboard device 1904 and receives content audibly via a speakerconfigured with smart glasses 1902 (e.g., speaker 453 of FIG. 4 ).Referring to FIG. 19 , processing device 460 (FIG. 4 ) may determinethat smart glasses 1902 are within the threshold distance and facingkeyboard device 1904 and may present the first response to the queryusing the first operation mode for smart glasses 1902. Accordingly,processing device 460 (FIG. 4 ) may visually present the weatherforecast on virtual display 1910, where the forecast may include themost probable forecast (e.g., topmost forecast), in addition to twoother alternative forecasts (e.g., bottom two forecasts). Referring toFIG. 23 , processing device 460 may determine that smart glasses 1902are within the threshold distance but face away from keyboard device1904. Processing device 460 may present the second response to the queryusing the second operation mode for smart glasses 1902. Accordingly,processing device 460 may audibly present the most probable forecast(“clouds and rain”) via a speakers configured with smart glasses 1902(e.g., speaker 453) and may omit presenting the alternative forecasts.

In some embodiments, the first response includes visually presenting ananswer to the query and additional relevant information, and the secondresponse includes audibly presenting the answer to the query withoutpresenting the additional relevant information. The term “answer” mayrefer to a response or information provided as feedback to a query. Theterm “additional relevant information” may refer to supplementary facts,advice, or data pertaining to the query, that may not be included in adirect or narrow response to a query. The term “without presenting theadditional relevant information” may refer to omitting or withholdingpresenting the additional information. For example, in response to aquery for directions, when the user is facing the keyboard device, adriving route may be displayed together with indications for servicestations along the route, and opening hours for the service stations.However, when the user is facing away from the keyboard device, thedriving route may be described audibly, and the information relating tothe service stations may be omitted.

By way of a non-limiting example, turning to FIGS. 19 and 23 , wearer1900 submits a query to receive a weather forecast by issuing a voicecommand “Open weather forecast”. Referring to FIG. 19 , smart glasses1902 are within the threshold distance and facing keyboard device 1904.Thus, processing device (FIG. 4 ) may apply the first operation mode anddisplay forecast weather app 1908 predicting rain. In addition,processing device may present wind speeds (e.g., additional relevantinformation) as text accompanying the rain forecast. Referring to FIG.23 , smart glasses 1902 are within the threshold distance but facingaway from keyboard device 1904. Thus, processing device may apply thesecond operation mode and audibly present the weather forecast (e.g.,clouds and rain) via speakers configured with smart glasses 1902 (e.g.,speakers 453), and may withhold presenting the wind speeds.

Some embodiments include determining that the user command is aninstruction to present a new virtual object, wherein in the firstoperation mode, the action responding to the instruction includespresenting the new virtual object in a first location associated with alocation of the keyboard device, and in the second operation mode, theaction responding to the instruction includes presenting the new virtualobject in a second location associated with a location of the wearableextended reality appliance. The term “instruction” may refer to adirective or order, e.g., to perform an action. The term “new virtualobject” may refer to a virtual (e.g., computer synthesized) item notpresented previously. Examples of a virtual object may include a virtualscreen, virtual widget, virtual icon or image, virtual application, orany other virtual item or entity. The term “location” may refer to aposition, e.g., relative to the keyboard device, work station, wearableextended reality appliance, or any other object of reference. Forexample, a user command may be a directive to open a new window orwidget for an application. When the user is facing the keyboard device,the new virtual object may be displayed docked, or relative to thekeyboard device (e.g., above or to the side of the keyboard device).However, when the user is facing away from the keyboard device, the newvirtual object may be displayed docked, or relative to the wearableextendable reality appliance.

By way of a non-limiting example, turning to FIGS. 19 and 21 ,processing device 460 (FIG. 4 ) may determine that the voice command“Open weather forecast” is an instruction to invoke a weatherapplication, requiring to display a new virtual window to display theweather forecast. Referring to FIG. 19 , smart glasses 1902 are withinthe threshold distance and facing keyboard device 1904. Thus, processingdevice may apply the first operation mode. Accordingly, processingdevice 460 may present forecast weather app 1908 as a new virtual objectinside a location of virtual screen 1910 associated with work station1906 and keyboard device 1904. Referring to FIG. 23 , smart glasses 1902are within the threshold distance but facing away from keyboard device1904. Thus, processing device may apply the second operation mode.Accordingly, processing device 460 may present forecast weather app 1908as a new virtual object in a location associated with (e.g., relativeto) smart glasses 1902.

In some embodiments, the first location is docked relative to thelocation of the keyboard device, and the second location changes withhead movements of a user of the wearable extended reality appliance. Theterm “docked” may refer to locked, anchored or tethered. For example, avirtual widget docked to a work station may be anchored to the workstation such that when a user leaves the work station, the virtualwidget is no longer visible. As another example, a virtual widget dockedto a wearable extended reality appliance may follow the gaze of thewearer. The term “relative to the location” may refer to with respectto, or as compared to the location. e.g., using the location as areference. The term “changes with head movements of a user of thewearable extended reality appliance” may refer to following or trackinghead motions (e.g., up/down, left/right, sideways) performed by thewearer of the wearable extended reality appliance. For example, avirtual widget displayed in a location that changes with head movementsof a user of the wearable extended reality appliance may be anchored tothe wearable extended reality appliance such that when the user turnshis head, the virtual widget remains within the field of view of theuser. Thus, when the wearer is facing the keyboard device, the virtualobject may be displayed in a manner that is anchored or fixed (e.g.,docked) relative to the keyboard device. When the wearer is facing awayfrom the keyboard device, the virtual object may be displayed in amanner that is anchored to the wearable extended reality appliance andfollows the gaze of the user as the user turns his head.

By way of a non-limiting example, referring to FIG. 19 , smart glasses1902 are within the threshold distance and facing keyboard device 1904.In response to the voice command “Open weather forecast”, processingdevice may apply the first operation mode and present forecast weatherapp 1908 as a new virtual object inside a location of virtual screen1910, that is tethered (e.g., fixed relative to) work station 1906 andkeyboard device 1904. Referring to FIG. 23 , while wearer 1900 is withinthe threshold distance of keyboard device 1904, wearer 1900 moves thehead to the left until smart glasses 1902 are facing away from keyboarddevice 1904. In response to the voice command “Open weather forecast”,processing device 460 may apply the second operation mode and presentforecast weather app 1908 as a new virtual object tethered to smartglasses 1902, e.g., moving leftwards following the directional gaze ofwearer 1900 as wearer 1900 moves the head leftwards. In a similarmanner, when wearer 1900 moves the head rightwards, smart glasses 1902may present forecast weather app 1908 in a manner that tracks the headmotion of wearer 1900, e.g., moving rightwards.

In the second operation mode, some embodiments further include:receiving image data captured using an image sensor included in thewearable extended reality appliance; analyzing the image data to detecta person approaching the user; and causing a modification to the secondlocation based on the detection of the person approaching the user. Insome examples, the image data may be analyzed using an object detectionalgorithm to detect the person. Further, the image data may be analyzedusing a motion tracking algorithm to determine that the detected personis approaching the user. In some examples, the image data may beanalyzed using a visual classification algorithm to determine that aperson is approaching the user. The term “image data captured using animage sensor” may refer to the detection of light signals by an imagesensor and conversion of the light signal to image pixels. The term“person approaching the user” may refer to another individual (e.g.,other than the user) moving towards the user who may be wearing thewearable extended reality appliance. The term “causing a modification tothe second location based on the detection of the person approaching theuser” may include changing or adjusting the position where content ispresented by accounting for the person approaching the user, e.g., sothat the content does not obstruct the person, or vice-versa. Forexample, the modification may be based on a direction from which theperson is approaching the user (for example, moving the second locationto another direction). In one example, the modification may be based ona distance of the approaching person (for example, avoiding themodification for far away persons). In one example, the modification maybe based on a direction of movement of the person (for example,estimating whether the person is indeed approaching the user based onthe direction of motion), e.g., when the person approaches from theleft, the location for presenting content may be shifted rightwards andthe reverse. In one example, the modification may be based on a speed ofthe person (for example, applying the modification when the person stopsmoving or moves very slowly). In one example, the modification may bebased on a gesture or a facial expression of the person (for example,when the person is gesturing to the user, the modification may beapplied), e.g., when the person is seeking the attention of the wearer,the location for presenting content may be shifted to the side.

By way of a non-limiting example, reference is now made to FIG. 24 whichis substantially similar to FIG. 22 with the notable difference of aperson 2400 approaching wearer 1900. A camera configured with smartglasses 1902 (e.g., image sensor 472 of FIG. 4 ) may capture an image ofperson 2400 approaching wearer 1900. Processing device 460 may analyzethe image to detect person 2400 approaching the wearer 1900 from theleft. For example, the position of person 2400 may overlap with and thusobstruct the display of forecast weather app 1908. Processing device 460may cause the location of forecast weather app 1908 to move (e.g.,shift) upwards and rightwards, based on the detected position of person2400, e.g., to prevent forecast weather app 1908 from obstructing person2400.

In the first operation mode, some embodiments further include: receivingimage data captured using an image sensor included in the wearableextended reality appliance; analyzing the image data to detect a surfacethat the keyboard device is placed on; and selecting the first locationbased on the detected surface. A surface may include the top layer of adesk, table, stool, a sliding tray (e.g., for a keyboard), or any otherflat, level area positioned as the top layer of an object. A surface maysupport another object resting on the surface such that the object isimmobile. For example, when the wearer is facing the keyboard device, anoptical sensor configured with the wearable extended reality appliancemay capture an image of the work station including a table supportingthe keyboard device. The image may be analyzed (e.g., by one or more ofprocessing devices 360 of FIG. 3, 460 of FIG. 4, 560 of FIG. 5 ) todetect the keyboard device resting on the surface of a desk. The newvirtual object instructed by the user command may be displayed based onthe surface of the desk. For example, the new virtual object may bedisplayed as a virtual widget resting on the desk, or inside a virtualscreen docked to the desk surface. In one example, the first locationmay be selected based on a position of an edge of the surface that thekeyboard device is placed on.

By way of a non-limiting example, turning to FIG. 19 , wearer 1900 isseated at work station 1906 adjacent to table surface 1912. A cameraconfigured with smart glasses 1902 (e.g., image sensor 472 of FIG. 4 )may capture an image of work station 1906 with keyboard device 1904resting thereon. Processing device 460 may analyze the image and detectkeyboard device 1904 resting on table surface 1912. Processing device460 may determine to apply the first operation mode for smart glasses1902 based on the distance and orientation of smart glasses 1902 tokeyboard device 1904. Processing device 460 may select the location todisplay forecast weather app 1908 based on table surface 1912, e.g., bydisplaying forecast 1908 inside virtual screen 1910 located just abovetable surface 1912. As another example, processing device 460 may selecta location on table surface 1912 to display a weather widget 1916.

Some embodiments further include determining the action for respondingto the identified user command based on the relative orientation of thewearable extended reality appliance with respect to the keyboard deviceand a posture associated with a user of the wearable extended realityappliance. The term “posture” may refer to a position for holding thebody. In some embodiments, the posture is selected from a groupincluding: lying, sitting, standing, and walking. The term “lying” mayrefer to reclining, e.g., horizontally. The term “sitting” may refer toa semi-upright position, resting the weight of the upper body on ahorizontal surface. The term “standing” may refer to a stationaryupright pose whereby the weight of the body is supported by the legs.The term “walking” may refer to an upright pose whereby the weight ofthe body is supported by the legs while in motion. For example, if thewearer is in an upright seated position such the wearable extendedreality appliance is substantially above the keyboard device, contentmay be displayed in a virtual screen located above the keyboard deviceusing a high resolution. However, if the wearer is reclining whilesitting next to the keyboard device (e.g., the wearer is facing theceiling), content may be displayed in a virtual screen against theceiling using a lower resolution.

By way of a non-limiting example, turning to FIG. 19 , based on thesitting posture and front-facing orientation of wearer 1900, processingdevice 460 (FIG. 4 ) may display forecast weather app 1908 in virtualscreen 1910 above keyboard device 1904 and directly in front of wearer1900. Turning to FIG. 20 , based on the walking posture of wearer 1900and relatively unstable (e.g., dynamic) orientation of smart glasses1902, processing device 460 may display forecast weather app 1908tethered to smart glasses 1902 in a manner that tracks the directionalgaze of wearer 1900.

Some embodiments further include determining the action for respondingto the identified user command based on the relative orientation of thewearable extended reality appliance with respect to the keyboard deviceand types of virtual objects displayed by the wearable extended realityappliance. The term “types of virtual objects” may refer to a categoryor characterization of a virtual item. For example, a virtual object maybe categorized according to context, time of presentation, duration ofpresentation, size, color, resolution, content type (e.g., text, image,video, or combinations thereof), resource demands (e.g., processing,memory, and or communications bandwidth), and any other characterizationof virtual objects. For example, the response to a user command maydepend on what additional virtual objects are currently displayed by thewearable extended reality appliance. When the wearer is seated at a deskadjacent to the keyboard device and is facing forwards, and the wearableextended reality appliance displays a calendar widget resting on thedesk top, in response to a request to view a meeting schedule, anupcoming meeting may be displayed via the calendar widget (e.g., by oneor more of processing devices 360 of FIG. 3, 460 of FIG. 4, 560 of FIG.5 ). When the wearer is seated at the desk but facing away from thekeyboard device, in response to a request to view the meeting schedule,the upcoming meeting may be displayed in a virtual window tethered tothe directional gaze of the wearer.

By way of a non-limiting example, turning to FIG. 19 , wearer 1900 isadjacent to and facing keyboard device 1904 while viewing content 1924via smart glasses 1902. In response to a request by wearer 1900 to opena weather forecast, processing device 460 (FIG. 4 ) may display forecastweather app 1908 to the side of content 1924, so as not to obstructcontent 1924.

Some embodiments provide a system for selectively operating wearableextended reality appliance, the system including at least one processorprogrammed to: establish a link between a wearable extended realityappliance and a keyboard device; receive sensor data from at least onesensor associated with the wearable extended reality appliance, thesensor data being reflective of a relative orientation of the wearableextended reality appliance with respect to the keyboard device; based onthe relative orientation, select from a plurality of operation modes aspecific operation mode for the wearable extended reality appliance;identify a user command based on at least one signal detected by thewearable extended reality appliance; and execute an action responding tothe identified user command in a manner consistent with the selectedoperation mode.

For example, turning to FIG. 19 , processing device 460 (FIG. 4 ) may beprogrammed to establish a link between smart glasses 1902 and keyboarddevice 1904. Processing device 460 may receive position data (e.g.,sensor data) from at least one sensor (e.g., GPS of motion sensor 473)associated with smart glasses 1902. The sensor data may be reflective ofa relative orientation of smart glasses 1902 with respect to keyboarddevice 1904. Based on the relative orientation, processing device 460may select from a plurality of operation modes (e.g., stored in memorydevice 411) a specific operation mode for smart glasses 1902 (e.g., todisplay content in virtual screen 1910). Processing device 460 mayidentify a user command based on at least one signal (e.g., an audiosignal) detected by audio sensor 471 of smart glasses 1902. Processingdevice 460 may execute an action responding to the identified usercommand in a manner consistent with the selected operation mode (e.g.,by displaying forecast weather app 1908 in virtual screen 1910).

FIG. 25 illustrates a block diagram of example process 2500 forinterpreting commands in extended reality environments based ondistances from physical input devices, consistent with embodiments ofthe present disclosure. In some embodiments, process 2500 may beperformed by at least one processor (e.g., processing device 460 ofextended reality unit 204, shown in FIG. 4 ) to perform operations orfunctions described herein. In some embodiments, some aspects of process2500 may be implemented as software (e.g., program codes orinstructions) that are stored in a memory (e.g., memory device 411 ofextended reality unit 204, shown in FIG. 4 ) or a non-transitorycomputer readable medium. In some embodiments, some aspects of process2500 may be implemented as hardware (e.g., a specific-purpose circuit).In some embodiments, process 2500 may be implemented as a combination ofsoftware and hardware.

Referring to FIG. 25 , process 2500 may include a step 2502 ofestablishing a link between a wearable extended reality appliance and akeyboard device. As described earlier, a communications channel may becreated between a wearable extended reality appliance and a keyboarddevice. For example, a Bluetooth channel may communicatively couple thewearable extended reality appliance with the keyboard device.

Process 2500 may include a step 2504 of receiving sensor data from atleast one sensor associated with the wearable extended realityappliance, the sensor data being reflective of a relative orientation ofthe wearable extended reality appliance with respect to the keyboarddevice. As described earlier, a sensor, such as an image sensor, amotion sensor, an IR sensor, and/or a radio sensor configured with thewearable extended reality appliance may sense data expressing therelative orientation of the wearable extended reality appliance and thekeyboard device.

Process 2500 may include a step 2506 of, based on the relativeorientation, selecting from a plurality of operation modes a specificoperation mode for the wearable extended reality appliance. As describedearlier, the relative orientation of the wearable extended realityappliance to the keyboard device may be used to choose a specificoperation mode from multiple candidate operation modes for wearableextended reality appliance.

Process 2500 may include a step 2508 of identifying a user command basedon at least one signal detected by the wearable extended realityappliance. As described earlier, the wearable extended reality appliancemay include a detector, such as a microphone (e.g., audio sensor 472 ofFIG. 4 ), an image sensor (e.g., image sensor 472), a motion sensor(e.g., motion sensor 472), an environmental sensor (e.g., environmentalsensor 474), and additional sensors (e.g., sensors 472). A processingdevice (e.g., one or more of processing devices 360 of FIG. 3, 460 ofFIG. 4, 560 of FIG. 5 ) may analyze the detected signal to identify auser command.

Process 2500 may include a step 2510 of executing an action respondingto the identified user command in a manner consistent with the selectedoperation mode. As described earlier, the action performed in responseto a user command may be performed in compliance with the operation modeof the wearable extended reality appliance (e.g., based on the relativeorientation to the keyboard device).

Videos of users wearing an extended reality appliance interacting withvirtual objects tend to depict the interaction from the perspective ofthe user. For example, a video may be from the perspective of the userwearing the extended reality appliance, such as a virtual depiction ofthe user's hands interacting with a virtual object.

An outside observer would only see the user's hands moving in thephysical environment while the user interacts with the virtual object(e.g., the observer would not be able to see the virtual object as theuser interacts with it). There is a desire to be able to generate avideo of the user interacting with the virtual object from theperspective of the outside observer, such that the outside observer maysee the virtual object as the user interacts with the virtual object.

Disclosed embodiments may include methods, systems, and non-transitorycomputer readable media for facilitating generating videos ofindividuals interacting with virtual objects. It is to be understoodthat this disclosure is intended to cover methods, systems, andnon-transitory computer readable media, and any detail described, evenif described in connection with only one of them, is intended as adisclosure of the methods, systems, and non-transitory computer readablemedia.

Some disclosed embodiments may be implemented via a non-transitorycomputer readable medium containing instructions for performing theoperations of a method. In some embodiments, the method may beimplemented on a system that includes at least one processor configuredto perform the operations of the method. In some embodiments, the methodmay be implemented by one or more processors associated with thewearable extended reality appliance. For example, a first processor maybe located in the wearable extended reality appliance and may performone or more operations of the method. As another example, a secondprocessor may be located in a computing device (e.g., an integratedcomputational interface device) selectively connected to the wearableextended reality appliance, and the second processor may perform one ormore operations of the method. As another example, the first processorand the second processor may cooperate to perform one or more operationsof the method. The cooperation between the first processor and thesecond processor may include load balancing, work sharing, or otherknown mechanisms for dividing a workload between multiple processors.

Some embodiments include a non-transitory computer readable mediumcontaining instructions for causing at least one processor to performoperations for generating videos of individuals interacting with virtualobjects. The terms “non-transitory computer readable medium,”“processor.” “instructions,” and “virtual objects” may be understood asdescribed elsewhere in this disclosure. As described below, one or moreprocessors may execute the one or more instructions for generating oneor more videos. As used herein, the term “video” may include a singlestill image, a series of one or more still images (e.g., a time lapsedsequence), or a continuous series of images (e.g., a video).

The one or more videos may illustrate one or more interactions of anindividual with one or more virtual objects. An individual may interactwith a virtual object in the extended reality environment in a similarmanner as the individual may interact with an object in the physicalenvironment. In some embodiments, any interaction that an individualcould have with an object in the physical environment may be replicatedin the extended reality environment. For example, the individual mayinteract with the virtual object by holding it in one or both of theirhands and may rotate the virtual object, by squeezing the virtualobject, by manipulating the virtual object, by looking at the virtualobject, by bringing the virtual object closer to themselves, or bymoving the virtual object farther away from themselves. In someembodiments, the individual may select one of several virtual objects byperforming a predefined hand gesture, for example “picking up” thevirtual object in the extended reality environment. In some embodiments,the individual may release a virtual object they are holding byperforming a predefined hand gesture, for example “dropping” or“tossing” the virtual object in the extended reality environment. Insome embodiments, the individual may interact with the virtual objectthrough voice commands or gesture commands. In one example, a gesturecommand for interacting with the virtual object may include virtuallytouching the virtual object. In another example, a gesture command forinteracting with the virtual object may be remote from the virtualobject and include no virtual touch.

Some embodiments include causing a wearable extended reality applianceto generate a presentation of an extended reality environment includingat least one virtual object. The terms “wearable extended realityappliance,” “extended reality environment,” and “virtual object” may beunderstood as described elsewhere in this disclosure. The presentationof the extended reality environment may be what the user sees whilewearing the extended reality appliance and may permit the user toperceive and/or interact with the extended reality environment. Theextended reality environment may be presented to the user of thewearable extended reality appliance by any of the mechanisms describedearlier.

Causing the wearable extended reality appliance to generate thepresentation of the extended reality environment may be performed by aprocessor associated with the wearable extended reality appliance. Insome embodiments, the processor may be a part of the wearable extendedreality appliance, such as the processing device 460 shown in FIG. 4 .In some embodiments, the wearable extended reality appliance may receivedata for display from a processor remote from and in communication withthe wearable extended reality appliance, such as from a computing deviceassociated with the wearable extended reality appliance. For example,the remote processor may include processing device 360 in input unit 202as shown in FIG. 3 . As another example, the remote processor mayinclude processing device 560 in remote processing unit 208 as shown inFIG. 5 .

The at least one virtual object may be, for example, at least onevirtual two-dimensional (2D) object (such as a virtual display screen, avirtual glass or other transparent surface, a virtual 2D graph, avirtual 2D presentation/slides, a 2D virtual user interface, etc.). Inanother example, the at least one virtual object may be at least onevirtual three-dimensional (3D) object (i.e., have volume), such as apuzzle cube, a ball, etc. In another example, the at least one virtualobject may include at least one virtual 2D object and at least onevirtual 3D object.

Some embodiments include receiving first image data from at least afirst image sensor, the first image data reflecting a first perspectiveof an individual wearing the wearable extended reality appliance. It isnoted that the phrase “user of the wearable extended reality appliance”and “individual wearing the wearable extended reality appliance” may beinterchangeable and for purposes of description herein may have asimilar meaning. Similarly, the short forms of these terms (e.g., “user”and “individual”) may also be interchangeable and have a similarmeaning.

Some embodiments include receiving image data (such as the first imagedata) from the wearable extended reality appliance. An “image sensor” asused herein may include a CCD sensor, a CMOS sensor, or any otherdetector capable of detecting images. Image data includes the output ofthe image sensor, or data derived or developed from the output of theimage sensor. In some embodiments, the image data may be received froman image sensor (e.g., the at least a first image sensor), such as a CCDsensor or a CMOS sensor located on or otherwise associated with thewearable extended reality appliance. Image data may be received viaeither a wired or wireless transmission, which transmission may be inthe form of digital signals.

For example, an image sensor 472 as shown in FIG. 4 may be employed inthe wearable extended reality appliance. The image data received from orcaptured by the image sensor (such as the first image data) may beassociated with the physical environment of the user and may include oneor more still images, a series of still images, or video.

In some embodiments, the first image data reflecting the firstperspective of the individual wearing the wearable extended realityappliance may represent what the individual would see (i.e., the “firstperspective”) when the extended reality environment is not beingdisplayed. For example, the first image data may capture an image of thephysical environment where the user is located while wearing theextended reality appliance, such as a room. In some embodiments, thefirst perspective may include items within the field of view of the userwhen the extended reality environment is not being displayed.

In some embodiments, the first image data may be stored in the wearableextended reality appliance. In some embodiments, the first image datamay be stored in a device separate from the wearable extended realityappliance and in wired or wireless communication with the first imagesensor such that the first data may be transmitted to the device, eitherupon initiation by the wearable extended reality appliance or by thedevice.

In some embodiments, the first image sensor is part of the wearableextended reality appliance. In some embodiments, the first image sensormay be considered to be “part” of the wearable extended realityappliance when it is physically attached to, physically embedded in, orotherwise associated with the wearable extended reality appliance. Forexample, the first image sensor may be physically attached to thewearable extended reality appliance via a physical connection mechanismsuch as a clip, a bracket, or a snap-fit arrangement. As anotherexample, the first image sensor may be physically attached to thewearable extended reality appliance via an adhesive. In someembodiments, the first image sensor may be located elsewhere on the userand may be in communication with the wearable extended realityappliance. For example, the first image sensor may be clipped orattached to the user's shirt or clothing.

In some embodiments, the first image sensor may be located on anexterior portion of the wearable extended reality appliance, such thatthe image sensor may be positioned to capture first image datacorresponding to the individual's head position. FIG. 26 illustrates anexemplary wearable extended reality appliance 2610 including a firstimage sensor 2612. While FIG. 26 illustrates wearable extended realityappliance 2610 as a pair of glasses, wearable extended reality appliance2610 may take on other forms (e.g., goggles) as described herein. Firstimage sensor 2612 is shown in FIG. 26 as being located to one side of aframe portion of wearable extended reality appliance 2610 (e.g., theleft side of the individual's head). In some embodiments, first imagesensor 2612 may be located on other portions of wearable extendedreality appliance 2610 without affecting the operation of first imagesensor 2612. For example, first image sensor 2612 may be located on theright side of wearable extended reality appliance 2610 relative to theindividual's head. As another example, first image sensor 2612 may belocated in the middle of wearable extended reality appliance 2610 suchthat first image sensor 2612 does not block the individual's visionthrough the lenses of wearable extended reality appliance 2610. In someembodiments, more than one image sensor 2612 may be located on theexterior portion of wearable extended reality appliance 2610. Forexample, a first image sensor may be located on the left side of theframe and a second image sensor may be located on the right side of theframe. As another example, a first image sensor may be located on theleft side of the frame, a second image sensor may be located on theright side of the frame, and a third image sensor may be located in themiddle of the frame.

FIG. 27 illustrates an exemplary view from the perspective of theindividual wearing the extended reality appliance (for example, ascaptured by first image sensor 2612). Image 2710 is an image of thephysical environment around the user and is captured from theperspective of the first image sensor (i.e., the perspective of theindividual wearing the extended reality appliance). Image 2710 includesa depiction 2712 of the user's hands (and may include a portion of theuser's arms) in the physical environment and a depiction of a computingdevice 2714 including an image sensor 2716. It is understood thatcomputing device 2714 and image sensor 2716 are provided for purposes ofillustration, and that other devices including one or more image sensorsmay also be used, or the device including the second image sensor maynot be depicted or may be only partly depicted in image 2710.

Some embodiments include receiving second image data from at least asecond image sensor, the second image data reflecting a secondperspective facing the individual. In some embodiments, the second imagesensor may have similar structural and/or functional characteristics asthe first image sensor described herein. The second image datareflecting the second perspective facing the individual wearing theextended reality appliance may be from a position in the physicalenvironment such that the second image sensor faces the individual. Insome embodiments, the second image sensor may be placed in any locationin the physical environment of the individual (and within an imagingrange of the second image sensor) such that the second image sensor maycapture one or more images of the individual.

In some embodiments, the second image sensor is a part of a computingdevice selectively connected to the wearable extended reality appliance.In some embodiments, the computing device may include an input device oran integrated computational interface device as described herein. Thecomputing device may be selectively connected to the wearable extendedreality appliance via a wired connection or a wireless connection asdescribed herein. In some embodiments, the second image sensor may beconsidered to be “part” of the computing device when it is physicallyattached to, physically embedded in, or otherwise associated with thecomputing device. For example, the second image sensor may be physicallyattached to the computing via a physical connection mechanism such as aclip, a bracket, or a snap-fit arrangement. As another example, thesecond image sensor may be physically attached to the computing devicevia an adhesive. As another example, the second image sensor may beembedded in a portion of a housing of the computing device. As anotherexample, the second image sensor may be associated with the computingdevice by being located separate from the computing device (such as in astandalone device) and in wired or wireless communication with thecomputing device.

FIG. 26 illustrates an exemplary computing device 2620 including asecond image sensor 2622. In some embodiments, computing device 2620 mayinclude more than one second image sensor. In some embodiments, secondimage sensor 2622 may be included in a standalone device (i.e., notselectively connected to computing device 2620 or to the wearableextended reality appliance), but may be configured to communicate withcomputing device 2620 or the wearable extended reality appliance viaeither wired or wireless communication. For example, second image sensor2622 may be included in a security camera in wireless communication withcomputing device 2620. In some embodiments, if a plurality of secondimage sensors are used, the second image sensors may each be located ina separate device, and each separate device may be in wired or wirelesscommunication with each other and/or with computing device 2620. Forexample, one second image sensor may be located in computing device 2620and another second image sensor may be located in a security cameraseparate from computing device 2620. As another example, a plurality ofsecurity cameras may be used, each security camera including a separatesecond image sensor.

In embodiments where the method is performed by a processor, theprocessor may be configured to receive the second image data from thesecond image sensor. In some embodiments, the second image data may bestored in a device that includes the second image sensor, such as thecomputing device or the separate device. In such embodiments, the secondimage data may be transmitted to the processor, for example, uponreceiving a command from the processor or on a periodic basis.

Some disclosed embodiments include obtaining image data from thecomputing device. In some embodiments, the image data may be obtainedfrom an image sensor, such as a CCD or CMOS sensor located on orotherwise associated with the computing device. For example, the imagesensor 372 as shown in FIG. 3 may be employed in the computing device.The image data received or generated by the image sensor may beassociated with the physical environment of the user and may include oneor more still images, a series of still images, or video.

FIG. 28 illustrates an exemplary view from the perspective of the secondimage sensor in the computing device, facing the individual wearing theextended reality appliance (i.e., from the second perspective). Image2810 includes a depiction 2812 of the user in the physical environmentand a depiction 2814 of the user's hands in the physical environment. Asshown in image 2810, while the user's hands may interact with a virtualobject, the virtual object cannot be seen from the second perspective.

FIG. 29 illustrates exemplary virtual objects, as seen from theperspective of the user of the wearable extended reality appliance.Image 2910 shows the virtual objects as seen in the extended realityenvironment. Image 2910 includes a virtual depiction 2912 of a puzzlecube, a virtual depiction 2914 of a volleyball, and a virtual depiction2916 of a vase of flowers. As shown in image 2910, the relative sizes ofvirtual depictions 2912, 2914, and 2916 may vary depending on thevirtual distance the virtual object is from the user. The term “virtualdistance,” as used herein, represents a distance between virtual objectsdisplayed in the extended reality environment, or a distance between theuser or the wearable extended reality appliance and a virtual objectdisplayed in the extended reality environment. Similar to the relativesizes of objects in the physical environment, using perspectivegeometry, a virtual object near the user may appear larger than avirtual object in the background. For example, if the user is holdingthe puzzle cube, virtual depiction 2912 may appear larger than virtualdepictions 2914 or 2916.

In some embodiments, the virtual objects may have “fixed” locations whenthe user is not interacting with the virtual object. As shown in image2910, the user is interacting with the puzzle cube and not thevolleyball or the vase of flowers. If the user were to change thevirtual object the user is interacting with, for example by virtuallyreleasing the puzzle cube and virtually picking up the vase of flowers,image 2910 may be updated to reflect that the user is now interactingwith the vase of flowers and not the puzzle cube. For example, thepuzzle cube may be placed in the same location that the vase of flowerswas in before the user virtually picked up the vase of flowers. In someembodiments, the virtual objects may change locations and/or appearance,even when the user is not interacting with the virtual object.

Some embodiments include identifying in the first image data firstphysical hand movements interacting with the at least one virtual objectfrom the first perspective. Referring to FIG. 27 , image 2710 (includingthe first image data) may be analyzed to identify user hand movements(i.e., the first physical hand movements) while interacting with thevirtual object. For example, the first physical hand movements may beidentified by performing an image recognition algorithm, a visualdetection algorithm, or a visual recognition algorithm, such as a visualactivity recognition algorithm or a visual gesture recognitionalgorithm. In some embodiments, the algorithm may include any machinelearning algorithm described earlier. For example, a machine learningmodel may be trained using training examples to identify hand movementsinteracting with virtual objects in images and/or videos. An example ofsuch training example may include a sample image and/or a sample videoof sample hands, together with a label indicating movements of thesample hands interacting with a sample virtual object. The trainedmachine learning model may be used to analyze the first image data andidentify the first physical hand movements interacting with the at leastone virtual object. In another example, the first image data may beanalyzed to calculate a convolution of the at least part of the firstimage data and thereby obtain a result value of the calculatedconvolution. Further, the identification of the first physical handmovements interacting with the at least one virtual object may be basedon the result value of the calculated convolution. For example, inresponse to the result value of the calculated convolution begin a firstvalue, the first physical hand movements may be identified asinteracting with the at least one virtual object, and in response to theresult value of the calculated convolution begin a second value, thefirst physical hand movements may be identified as not interacting withany virtual object. In embodiments where the method is performed by aprocessor, the processor may be configured to perform one or more ofthese algorithms to identify the first physical hand movements. In someembodiments, the algorithms to identify the first physical handmovements may be performed by a specialized processor in communicationwith the processor performing the method, for example, by a graphicsprocessing unit (GPL), an application specific integrated circuit(ASIC), a digital signal processor (DSP), or a field programmable gatearray (FPGA).

Some embodiments include identifying in the second image data secondphysical hand movements interacting with the at least one virtual objectfrom the second perspective. Referring to FIG. 28 , image 2810 may beanalyzed to identify user hand movements (i.e., the second physical handmovements) while interacting with the virtual object. In someembodiments, the second physical hand movements may be identified in asimilar manner as the first physical hand movements.

In some embodiments, the first physical hand movements and the secondphysical hand movements may be the same physical hand movements, butfrom different perspectives (i.e., from the first perspective and thesecond perspective). In some embodiments, the first physical handmovements and the second physical hand movements may be differentphysical hand movements. For example, the first physical hand movementsand the second physical hand movements may be captured at differentmoments in time.

Some embodiments include analyzing at least one of the first image dataor the second image data to determine an interaction with the at leastone virtual object. Analyzing the first image data and/or the secondimage data to determine an interaction with the virtual object mayinclude performing an image recognition algorithm, a visual detectionalgorithm, or a visual recognition algorithm, such as a visual activityrecognition algorithm or a visual gesture recognition algorithm. In someembodiments, the algorithm may include any machine learning algorithmdescribed earlier. For example, a machine learning model may be trainedusing training examples to determine interaction with virtual objects.An example of such training example may include a sample image and/or asample video, together with a label indicating interaction with a samplevirtual object. The trained machine learning model may be used toanalyze the first image data and/or the second image data and identifythe interaction with the at least one virtual object. In anotherexample, the first image data may be analyzed to calculate a convolutionof the at least part of the first image data and thereby obtain a firstresult value of the calculated convolution. Further, the second imagedata may be analyzed to calculate a convolution of the at least part ofthe second image data and thereby obtain a second result value of thecalculated convolution. Further, the identification of the firstphysical hand movements interacting with the at least one virtual objectmay be based on the first result value of the calculated convolution andthe second result value of the calculated convolution. For example, inresponse to a first combination of the first and second result values,an interaction with the at least one virtual object may be determined,and in response to a second combination of the first and second resultvalues, no interaction with the at least one virtual object may bedetermined. In embodiments where the method is performed by a processor,the processor may be configured to perform the one or more algorithms todetermine the interaction. In some embodiments, the algorithms todetermine the interaction may be performed by a specialized processor incommunication with the processor performing the method, for example, bya graphics processing unit (GPU), an application specific integratedcircuit (ASIC), a digital signal processor (DSP), or a fieldprogrammable gate array (FPGA).

In some embodiments, the analyzing may include analyzing both the firstimage data and the second image data. In some embodiments, theinteraction may correspond only to the first physical hand movements. Insome embodiments, the interaction may correspond only to the secondphysical hand movements. In some embodiments, the interaction maycorrespond to both the first physical hand movements and the secondphysical hand movements.

The interaction with the virtual object may include any interaction theuser may have with a virtual object in the extended reality environment.The scope of the possible interactions with the virtual object maymirror the scope of possible interactions an individual may have with acorresponding physical object in the physical environment. For example,if the user is holding the virtual puzzle cube, the possibleinteractions include any interaction the user may have with a physicalpuzzle cube in the physical environment, such as turning the entirepuzzle cube in their hands, turning one portion of the puzzle cube withone hand while holding the rest of the puzzle cube with the other hand,picking up the puzzle cube, putting the puzzle cube on a surface,throwing the puzzle cube, or handing the puzzle cube to another person.As another example, if the user is holding a virtual volleyball, thepossible interactions include any interaction the user may have with aphysical volleyball in the physical environment, such as turning thevolleyball in their hands, hitting the volleyball from a variety ofdifferent hand positions (e.g., serving, bumping, setting, or spiking),throwing the volleyball, or handing the volleyball to another person.

As the user interacts with the virtual object in the extended realityenvironment, the appearance of the virtual object may change. Forexample, if the user is holding the virtual volleyball and rotates thevolleyball away from themselves with both hands, it may appear to theuser in the extended reality environment that both hands are rotatingaway from themselves while the volleyball is also rotating. In someembodiments, the changes in the extended reality environment may bedetected by an input device as described herein, such as a pair ofhaptic gloves. To properly display these changes to the user, thepresentation of the extended reality environment may be updated toreflect these changes. While such changes are occurring in thepresentation of the extended reality environment, the first image sensorand the second image sensor may capture the first image data and thesecond image data of the user's hands rotating away from the user's body(i.e., the first physical hand movements and the second physical handmovements).

In some embodiments, determining the interaction with the virtual objectmay include determining how the appearance of the virtual object ischanged based on the user's interaction. For example, if the user isholding the virtual puzzle cube and the interaction is that the userturns the top portion of the puzzle cube in a counterclockwisedirection, in the extended reality environment, the user would see thetop portion of the puzzle cube turning in the counterclockwisedirection.

Some embodiments include rendering for display a representation of theat least one virtual object from the second perspective. So that thevirtual object may be properly seen from the second perspective (i.e.,as the virtual object may appear to an outside observer as if theoutside observer is “looking into” the extended reality environment), itneeds to be rendered (e.g., generated, drawn, illustrated, pictured,shown, represented, or presented) from the second perspective. In theextended reality environment, the virtual object may be rendered (e.g.,generated as part of the presentation of the extended realityenvironment) from the user's perspective. The virtual object maytherefore be shown (or generated for presentation to the user) from thesecond perspective (e.g., that of an external observer “looking into”the extended reality environment).

For example, if the user is holding the virtual puzzle cube, the usermay see a certain color combination facing in the user's direction inthe extended reality environment. To render the puzzle cube from thesecond perspective (i.e., to render the face of the puzzle cube that theoutside observer would see if the user were holding the puzzle cube inthe physical environment), the rendering may include using ray castingalgorithms, artificial intelligence (AI) algorithms, machine learning(ML) algorithms, 3D models of the puzzle cube, and/or information fromthe wearable extended reality device (e.g., what the user sees in theextended reality environment) about the puzzle cube. For example, thealgorithms may use information about the virtual object to render thevirtual object from any angle (i.e., from the first perspective or thesecond perspective). Then, as the user interacts with the virtualobject, changing what the user sees in the extended reality environment(i.e., from the first perspective), the algorithm may correspondinglyupdate how the virtual object appears from other angles (i.e., from thesecond perspective). In some embodiments, the rendering may be based onstored views of the virtual object from various angles and may select aview that represents an opposite side of the virtual object from whatthe user is currently viewing in the extended reality environment. Insome embodiments, the information about the virtual object may be storedin a database or other data storage that may be accessed as part of therendering.

Some embodiments include melding the rendered representation of the atleast one virtual object from the second perspective with the secondimage data to generate a video of the individual interacting with the atleast one virtual object from the second perspective. As noted above,the rendered representation of the virtual object from the secondperspective is what the outside observer would see if they could “lookinto” the extended reality environment. The melding may include aprocess of combining the rendered virtual object from the secondperspective with the second image data (for example, similar to a “greenscreen” or “chroma key” effect in television or movies where one imageis layered or composited with a second image). In some examples, themelding may include image stitching and/or object blending algorithms.In some examples, the melding may include using a generative modelanalyze the rendered representation of the at least one virtual objectfrom the second perspective and the second image data to generate thevideo of the individual interacting with the at least one virtual objectfrom the second.

In embodiments where the method is performed by a processor, theprocessor may be configured to perform the melding, such as byperforming a chroma keying algorithm. In some embodiments, the meldingmay be performed by a specialized processor in communication with theprocessor performing the method, for example, by a graphics processingunit (GPU), an application specific integrated circuit (ASIC), a digitalsignal processor (DSP), or a field programmable gate array (FPGA).

In some embodiments, generating the video of the individual interactingwith the virtual object from the second perspective may be performed bycombining several melded still images together. In some embodiments,generating the video may be performed by melding a video of the renderedrepresentation of the virtual object from the second perspective with avideo of the individual from the second perspective. In embodimentswhere the method is performed by a processor, the processor may beconfigured to generate the video. In some embodiments, the video may begenerated by a specialized processor in communication with the processorperforming the method, for example, by a graphics processing unit (GPU),an application specific integrated circuit (ASIC), a digital signalprocessor (DSP), or a field programmable gate array (FPGA).

In some embodiments, the generated video may be stored for laterplayback. For example, the generated video may be stored on thecomputing device. As another example, the generated video may be storedon a separate storage remote from the wearable extended realityappliance and the computing device, such as on a cloud-based storageservice. In some embodiments, the generated video may not be stored andmay be displayed on a screen (e.g., a television, a monitor, a tablet, amobile phone, a mobile device, or other display device).

FIG. 30 illustrates an exemplary melded view from the perspective of thesecond image sensor. As illustrated in the figure, melded image 3010includes an individual wearing extended reality appliance 3012 andinteracting with a virtual object 3016. Melded image 3010 also includesa depiction 3014 of the user's hands, in the physical environment fromthe second perspective, holding a virtual object 3016 (shown as a puzzlecube). In some embodiments, melded image 3010 may include a still image,a series of one or more still images, or a video. By melding therendered representation of the virtual object from the secondperspective with the second image data (i.e., the video captured fromthe second perspective), the generated video may represent what theoutside observer sees as the user interacts with the virtual object.

In some embodiments, the identified first physical hand movements areassociated with a gesture for causing a movement of the at least onevirtual object. A gesture for causing a movement of the virtual objectmay include any hand motion that the user may make to move the virtualobject in the extended reality environment. For example, the user mayturn the virtual object (i.e., a movement of the virtual object) intheir hands by rotating one or both of their hands in the direction thatthey wish to turn the virtual object (i.e., physical hand movementsassociated with a gesture). In some embodiments, the identified secondphysical hand movements may be associated with a gesture for causing themovement of the at least one virtual object. In some embodiments, boththe identified first physical hand movements and the identified secondphysical hand movements may be associated with a gesture for causing themovement of the at least one virtual object. The gesture may causemovement of a virtual object as the result of the gesture beingrecognized through machine vision as having a particular function. Forexample, image analysis performed on hand motion may detect a type ofrotation that corresponds in memory to a particular object movement, andthat movement may then be translated to the associated virtual object.

In some embodiments, rendering the representation of the at least onevirtual object from the second perspective reflects the movement of theat least one virtual object. As the virtual object is moved in theextended reality environment, the rendered representation of the virtualobject from the second perspective is updated to reflect the movement.The rendering to reflect the movement of the virtual object may beperformed in a similar manner as described above. For example, as shownin FIG. 30 , if the user turns one portion of the virtual puzzle cubeaway from the user, the rendered representation of the virtual puzzlecube from the second perspective would show that the portion of thevirtual puzzle cube is turned toward the outside observer.

In some embodiments, the user may interact with the virtual object inways other than by moving the virtual object. For example, the userinteraction may include changing the virtual object's size bysimultaneously moving both hands toward the virtual object to make thevirtual object smaller or by simultaneously moving both hands away fromthe virtual object to make the virtual object larger. As anotherexample, the user may change the orientation of the virtual object byturning it. As another example, the user may change the location of thevirtual object in the extended reality environment by moving the virtualobject from one portion of the extended reality environment to anotherportion of the extended reality environment. For example, the user maymove the virtual object from the left side of the extended realityenvironment to the right side of the extended reality environment.

As another example, the user may change the appearance of the virtualobject by changing the color of the virtual object. In some embodiments,the user may change the appearance of the virtual object by using a userinterface tool. For example, to change the color of a virtual object,the user may use a color picker tool.

In some embodiments, the user may change information presented by thevirtual object. For example, if the virtual object is a virtual screencontaining a text document, the user may scroll through the text, mayadd text, or may delete text, thereby changing the information presentedon the virtual screen. In such an example, the rendered representationof the virtual screen may include a “backwards” version of the textdocument such that the text orientation is the opposite of what the usersees in the extended reality environment (i.e., the renderedrepresentation may appear to the outside observer as if the outsideobserver was looking through a transparent screen of text from behind).

In some embodiments, the identified first physical hand movements areassociated with a gesture for causing a modification to a visualappearance of a portion of a surface of the at least one virtual object.For example, the identified first physical hand movements may beassociated with a predefined gesture (i.e., the user moves their handsin a predefined way or to a predefined location in the extended realityenvironment) to activate a user interface tool to enable the user tochange a color of a portion of the surface of the virtual object. Forexample, if the user moves their left hand to the upper left corner ofthe extended reality environment, a user interface tool panel may beactivated (i.e., appear in the extended reality environment) from whichthe user may select a tool with which they may interact with the virtualobject. For example, the user may use a color picker tool to change thecolor of a virtual object. As another example, the user may sizechanging tool to change the size of the virtual object. As anotherexample, the user may use a text tool to add text to the virtual object.As another example, the user may use one or more drawing tools, such asa predefined shape tool (e.g., a square, a rectangle, or a circle) or afreehand drawing tool (e.g., a pencil, a marker, or a paintbrush), todraw on the virtual object. In some examples, the at least one virtualobject may include a user interface, and the gesture for causing themodification to the visual appearance of the portion of the surface ofthe at least one virtual object may include at least one of gesture forentering data into the user interface, gesture for selecting an elementof the user interface, gesture for minimizing at least one element ofthe user interface, or gesture for expanding at least one element of theuser interface.

In some embodiments, the portion of the surface is visible from thesecond perspective and is not visible from the first perspective. Forexample, if the virtual object is a volleyball, the user may change thecolor of one panel of the volleyball (i.e., a portion of the surface ofthe volleyball) and that one panel may be visible from the outsideobserver's perspective (i.e., the second perspective) but not from theuser's perspective (i.e., the first perspective).

In some embodiments, rendering the representation of the at least onevirtual object from the second perspective reflects the modification tothe visual appearance of the surface of the at least one virtual object.The rendering to reflect the modification to the visual appearance ofthe surface of the virtual object may be performed in a similar manneras described above. The rendering would show the modified appearance ofthe surface of the virtual object as seen from the perspective of theoutside observer.

In some embodiments, the modification to the visual appearance of theportion of the surface of the virtual object may be visible to the userinteracting with the virtual object in the extended reality environmentbut may not be visible from the second perspective (i.e., to the outsideobserver) in the generated video. For example, if the virtual object isa volleyball, the user may change the color of one panel of thevolleyball (i.e., a portion of the surface of the volleyball) and thatone panel may be visible from the user's perspective in the extendedreality environment (i.e., the first perspective) but not from theoutside observer's perspective (i.e., the second perspective).

In some embodiments, the operations include determining a position ofthe at least one virtual object in the extended reality environment. Insome embodiments, determining the position of the virtual object may bebased on a location of the virtual object relative to a fixed locationin the extended reality environment (for example, relative to the user'slocation in the extended reality environment). For example, in theextended reality environment, the virtual object may not be in theuser's hands but may be located at a distance from the user (i.e., theuser would have to reach to hold the virtual object). In someembodiments, determining the position of the virtual object may be basedon a location of the virtual object relative to one or more othervirtual objects. In some embodiments, the position of the virtual objectmay be determined based on a distance from a fixed location or on adistance from a predetermined coordinate position. For example, theextended reality environment may include an internal coordinate systemand the position of a virtual object may be determined based on thelocation of the virtual object in that coordinate system. In someembodiments, the coordinate system may include a field-of-view of theextended reality environment, as described elsewhere in this disclosure.In some embodiments, the coordinate system may include an entirety ofthe extended reality environment, including portions of the extendedreality environment outside the field-of-view.

In some embodiments, rendering the representation of the at least onevirtual object from the second perspective is based on the determinedposition. The rendering of the virtual object from the secondperspective based on the determined position may be performed in asimilar manner as described above. Continuing the above example, thevirtual object may be rendered from the second perspective as beingpositioned at a distance from the user. For example, from the secondperspective, the virtual object may appear to be closer than to the user(i.e., the virtual object may appear to be located between the outsideobserver and the user).

In some embodiments, the determined position of the at least one virtualobject includes a distance between the at least one virtual object andthe individual. For example, in the extended reality environment, thevirtual object may appear to be 0.5 meters away from the individual. Insome embodiments, the distance may be any distance and may be measurablein any units, such as, but not limited to, millimeters, centimeters,meters, inches, or feet.

In some embodiments, rendering the representation of the at least onevirtual object from the second perspective is based on the determineddistance. The rendering of the virtual object from the secondperspective based on the determined distance may be performed in asimilar manner as described above. Since objects appear smaller atgreater distances, the determined distance may impact the size of theobject rendered. Similarly, the determined distance may also impactperspective, and the rendered perspective may reflect the determineddistance. Continuing the above example, if the virtual object is 0.5meters away from the individual in the extended reality environment,then from the second perspective, the virtual object may also berendered to be 0.5 meters away from the individual.

As another example, in the extended reality environment, the user maymove the virtual object toward the user or away from the user. So, ifthe user moves the virtual object toward herself, the virtual object mayappear to the user in the extended reality environment as gettinglarger, whereas from the second perspective, the virtual object mayappear to the outside observer as getting smaller. Similarly, if theuser moves the virtual object away from herself in the extended realityenvironment, the virtual object may appear to the user in the extendedreality environment as getting smaller, whereas from the secondperspective, the virtual object may appear to the outside observer asgetting larger.

In some embodiments, the determined position of the at least one virtualobject includes a spatial orientation of the at least one virtual objectwithin the extended reality environment. The spatial orientation of thevirtual object within the extended reality environment relates to theposition, attitude, inclination, and/or rotation of the virtual objectin the extended reality environment relative to other objects in theextended reality environment, including the user. For example, if theextended reality environment includes multiple virtual objects indifferent locations, the relative orientation of each virtual object(i.e., the spatial orientation of each virtual object relative to eachother) may be determined. As another example, the spatial orientation ofa virtual object may be determined based on a predetermined coordinateposition. For example, the extended reality environment may include aninternal coordinate system and the spatial orientation of a virtualobject may be determined based on the spatial orientation of the virtualobject in that coordinate system. In some embodiments, the coordinatesystem may include a field-of-view of the extended reality environment,as described elsewhere in this disclosure. In some embodiments, thecoordinate system may include an entirety of the extended realityenvironment, including portions of the extended reality environmentoutside the field-of-view. In some examples, the spatial orientation ofthe at least one virtual object may be selected to make a selected sideof the at least one virtual object to face the user. For example, theselected side may include textual data, and the selection of the spatialorientation may make the textual data viewable (and in some casesreadable) by the user. In some examples, the spatial orientation of theat least one virtual object may be selected to make a selected side ofthe at least one virtual object be directed to a selected direction inthe extended reality environment. For example, a spatial orientation forvirtual vase of flowers 2916 may be selected so that the opening of thevirtual vase is facing up. In another example, a spatial orientation ofa virtual arrow may be selected so that the virtual arrow points to aselected location and/or direction.

In some embodiments, rendering the representation of the at least onevirtual object from the second perspective is based on the determinedspatial orientation. The rendering of the virtual object from the secondperspective based on the determined spatial orientation may be performedin a similar manner as described above. Continuing the above example,the relative position of each virtual object as viewed in the extendedreality environment may be maintained when the virtual objects arerendered from the second perspective. As another example, the positionof each virtual object when rendered from the second perspective may bemaintained based on the location and/or spatial orientation of thevirtual object in a coordinate system as described above.

Referring to FIG. 29 , image 2910 includes virtual depiction 2912 of apuzzle cube, virtual depiction 2914 of a volleyball, and virtualdepiction 2916 of a vase of flowers. Image 2910 is from the firstperspective and shows how virtual objects 2912, 2914, and 2916 mayappear to a user while viewing the extended reality environment. Asshown in image 2910, the spatial orientation of the virtual objects isthat the vase of flowers 2916 appears to the user's left, the puzzlecube 2912 appears in front of the user or in the user's hands (i.e., inthe center of the extended reality environment), and the volleyball 2914appears to the user's right.

FIG. 31 is a melded image 3110 from the second perspective. Image 3110includes a depiction 3112 of the user in the physical environment, adepiction 3114 of the user's hands in the physical environment holding avirtual object 3116 of a puzzle cube, a depiction 3118 of a volleyball,and a depiction 3120 of a vase of flowers. As can be seen by comparingimage 2910 and image 3110, the relative positions of the virtual objects(i.e., the spatial orientation) from the first perspective (as shown inimage 2910) are maintained in the second perspective (as shown in image3110). From the second perspective as shown in image 3110, the spatialorientation of the volleyball 3118 appears to the outside observer'sleft, the puzzle cube 3116 appears in front of the user or in the user'shands (i.e., in the center of image 3110), and the vase of flowers 3120appears to the outside observer's right.

In some embodiments, the at least one virtual object includes textpresented by the wearable extended reality appliance on a side of the atleast one virtual object that faces the individual. In some embodiments,the text may be presented as a layer on top of the virtual object suchthat part of the virtual object is occluded. In some embodiments, “text”as used herein may also include an image, a logo, a text message,instructions, icons, arrows, or an alert.

In some embodiments, generating the video includes providing arepresentation of the text in the video. Text may include characters,words, sentences, lettering, symbols, or any other form of expression. Arepresentation of text may include an illustration or presentation ofthe particular form of expression from an associated perspective. Forexample, if the virtual object is transparent or intended to betransparent (e.g., a virtual screen including a text document), thegenerated video may include a representation of the text appearing as a“backwards” image of what the user sees (i.e., the text may appear tothe outside observer to be in the opposite orientation). For example, ifthe text appears to the user in the extended reality environment (i.e.,from the first perspective) in a left-to-right orientation, therepresentation of the text in the generated video may appear to theoutside observer (i.e., from the second perspective) in a right-to-leftorientation.

In some embodiments, the text may only be visible to the individual inthe extended reality environment and may not be visible to the outsideobserver (i.e., may be visible from the first perspective and may not bevisible from the second perspective). For example, if the virtual objectis a solid object (e.g., a coffee mug) with text on one side facing theindividual in the extended reality environment, the outside observerwould not be able to see through the coffee mug to read the text fromthe second perspective. This is the same result as would occur in thephysical environment with a physical coffee mug (i.e., in the physicalenvironment, the outside observer cannot see through the physical coffeemug to read the text facing the individual).

In some embodiments, rendering the representation of the at least onevirtual object includes determining an opacity for the representation ofthe at least one virtual object. The opacity of the virtual objectrepresents how transparent the virtual object is and whether the usercan “see through” the virtual object to be able to observe the physicalenvironment. In some embodiments, the opacity of the virtual object maybe automatically adjusted based on detection of activity in the physicalenvironment. For example, if the outside observer approaches theindividual wearing the extended reality appliance, the opacity of one ormore virtual objects (or of the entire extended reality environment) maybe adjusted such that the individual can see the outside observer in thephysical environment.

For example, if the virtual object is a virtual screen including text,the opacity of the virtual screen may be determined such that theoutside observer may see the text when rendered from the secondperspective; i.e., the opacity may be determined to be low enough thatthe outside observer may see through the virtual object when renderedfrom the second perspective. In some embodiments, the opacity isdetermined to be less than 75% when the at least one virtual objectobscures at least a portion of the individual from the secondperspective. For example, if the virtual object is a solid object (i.e.,not meant to be transparent), then the opacity of the virtual objectwhen rendered from the second perspective may be reduced (e.g., to 75%or less) such that the individual may be visible “through” the virtualobject even if the individual may not be fully visible. For example, theopacity of the virtual object may be reduced such that the outsideobserver may be able to see through the virtual object to see theindividual. As another example, the opacity of the virtual object may bereduced such that the outside observer may be able to see through thevirtual object to read text or see an image on a side of the virtualobject that faces the individual in the extended reality environment.

In some embodiments, the at least one virtual object contains privateinformation and rendering the representation of the at least one virtualobject includes obscuring the private information. Private informationmay include any data or representation designated as confidential to oneor more persons, or otherwise restricted for viewing purposes. Therendering of the virtual object to obscure the private information maybe performed in a similar manner as described above. For example, thevirtual object may include a virtual screen with a text document and thetext document may contain private information. In some embodiments, thefact that the virtual object contains private information may beindicated by a private information identifier, such as a flag or othertype of identifier. If the private information identifier is detected(e.g., the private information identifier is set or otherwise indicatesthat the virtual object contains private information), then renderingthe virtual object from the second perspective may include obscuring theprivate information. For example, the rendered virtual object mayinclude “greeked” text (e.g., rendering of the text as unreadablesymbols or lines) such that the private information is not readable fromthe second perspective. As another example, the opacity of the virtualobject may be adjusted such that the outside observer cannot see throughthe virtual object from the second perspective to read the privateinformation. As another example, the text may be obscured by othermeans, such as blurring or distorting the text, covering the text withan opaque box, or changing the color of the text to match thebackground. In any of these examples, the user in the extended realityenvironment would still be able to read the private information.

In some embodiments, the at least one virtual object includes a firstobject in a position visible from the second perspective and a secondobject in a position hidden from the second perspective. For example, inthe extended reality environment, the second virtual object may appearto be closer to the user and the first virtual object may appear to bebehind the second virtual object. In some embodiments, the first virtualobject may be larger than the second virtual object such that the firstvirtual object may be partially obscured by the second virtual object inthe extended reality environment. From the second perspective, the firstvirtual object may be visible to the outside observer, but the secondvirtual object may be blocked from view by the first virtual object(e.g., if the first virtual object is larger than the second virtualobject) such that the outside observer cannot see the second virtualobject.

In some embodiments, the generated video includes a representation ofthe first object from the second perspective in a first visual formatand a representation of the second object from the second perspective ina second visual format, the second visual format differs from the firstvisual format and is indicative of the second object being in theposition hidden from the second perspective. Generating the video may beperformed in a similar manner as described earlier. In some embodiments,different virtual objects may be rendered in different visual formats. Avisual format may include parameters of how a virtual object is to berendered, for example, object size, line width, line color, line style,object fill, object fill color, intensity, texture, ortransparency/opacity. In some embodiments, the different visual formatsmay include an indicator or a parameter (e.g., a flag, a tag, or otherindicator) in the visual format that the virtual object is in a hiddenposition. In some examples, the different visual formats may beindicative of the virtual object being in a hidden position. Forexample, a half transparent rendering of a normally non-transparentvirtual object may be indicative of the virtual object being in a hiddenposition.

For example, in the extended reality environment, the first virtualobject may be larger than the second virtual object and may be behindthe second virtual object such that from the second perspective thesecond virtual object may be hidden behind the first virtual object(i.e., the outside observer cannot see the second virtual object). Torender the image from the second perspective such that the outsideobserver may see both the first virtual object and the second virtualobject, the first virtual object may be rendered with a reduced opacity(i.e., the first visual format) and the second virtual object may berendered with its “normal” opacity (i.e., the second visual format) suchthat the outside observer may see “through” the first virtual object tosee the second virtual object. As another example, the first virtualobject may be rendered in an outline representation (i.e., not “filledin” or as a “wireframe” in the first visual format) and the secondvirtual object may be rendered with its “normal” opacity (i.e., thesecond visual format) such that the outside observer may see “through”the first virtual object to see the second virtual object.

In some embodiments, in the generated video the rendered representationof the at least one virtual object from the second perspective hides thephysical hand and includes a virtual representation of the physicalhand. Generating the video may be performed in a similar manner asdescribed earlier. For example, the generated video may include avirtual representation of the user's hands instead of a representationof the user's hands in the physical environment. The virtualrepresentation of the user's hands from the second perspective may berendered in a similar manner as rendering the virtual object from thesecond perspective. In some embodiments, the virtual representation ofthe user's hands may be based on the first image data including theuser's hands in the physical environment from the first perspective andthe second image data including the user's hands in the physicalenvironment from the second perspective. In some embodiments, thevirtual representation of the user's hands from the second perspectivemay be generated using a machine learning algorithm, such as agenerative adversarial network (GNN), a convolutional neural network(CNN), a recurrent neural network (RNN), or other machine learningalgorithm as described earlier. In some embodiments, the generated videomay include a virtual representation of the user's hands and a generatedrepresentation of the user's face without the wearable extended realityappliance.

In some embodiments, the operations include analyzing at least one ofthe first image data or the second image data to determine an absence ofinteraction with a particular virtual object. Analyzing the first imagedata or the second image data may be performed in a similar manner asdescribed earlier. In some embodiments, in the extended realityenvironment, the user may have stopped interacting with the virtualobject. For example, the user may place the virtual object on a surfaceor drop the virtual object. In some embodiments, when a user “drops” thevirtual object in the extended reality environment, the virtual objectmay “float” in front of the user until the user moves the virtual objectto a different location. In some embodiments, when a user “drops” thevirtual object in the extended reality environment, the virtual objectmay automatically be placed in a predetermined location.

In some embodiments, when the user stops interacting with the virtualobject, the generated video excludes a representation of the particularvirtual object. Generating the video may be performed in a similarmanner as described above. For example, if the user stops interactingwith the virtual object, the user may be able to see the virtual objectin the extended reality environment, but the outside observer would nolonger see the virtual object in the generated video. In someembodiments, however, when the user stops interacting with the virtualobject, the generated video may include the representation of theparticular virtual object. For example, the virtual object may “float”near the user from the second perspective. As another example, thevirtual object may be automatically placed in a predetermined location,visible both in the extended reality environment and in the generatedvideo from the second perspective.

In some embodiments, the operations include rendering for display arepresentation of the extended reality environment from the secondperspective. The rendering may be performed in a similar manner asdescribed earlier. For example, the entire extended reality environmentas seen by the user wearing the extended reality appliance may berendered from the second perspective; i.e., the outside observer may seeeverything that the user sees, but from the second perspective, similarto what the outside observer would see if they wore an extended realityappliance and were viewing the same extended reality environment as theuser.

In some embodiments, the operations include generating an additionalvideo of the individual in the extended reality environment interactingwith the at least one virtual object from the second perspective.Generating the additional video may be performed in a similar manner asdescribed earlier. In some embodiments, the additional video may includea complete representation of the extended reality environment meldedwith a representation of the user in the physical environment. Forexample, in the generated video, it may appear that the extended realityenvironment is placed between the outside observer and the individual inthe physical environment. In some embodiments, the additional video mayinclude a complete representation of the extended reality environmentmelded with a virtual representation of the individual in the extendedreality environment. For example, in the generated video, it may appearto the outside observer as if the outside observer was wearing anextended reality appliance and viewing the same extended realityenvironment as the individual.

In some embodiments, the operations include artificially deleting thewearable extended reality appliance from the second image data.Artificially deleting the wearable extended reality appliance from thesecond image data may enable the outside observer to see the user'sentire face, as if the user were not wearing the extended realityappliance. In some embodiments, the artificially deleting may beperformed in a similar manner as the rendering described earlier.

In some embodiments, the operations include generating the video of theindividual, without the wearable extended reality appliance, interactingwith the at least one virtual object from the second perspective.Generating the video may be performed in a similar manner as describedabove. In some embodiments, an image of the individual's face from thesecond perspective may be replaced with a previously captured image ofthe individual's face taken while the individual was not wearing theextended reality appliance. The previously captured image of theindividual's face may be retrieved from a storage location. For example,the storage location may be on the wearable extended reality appliance,on the computing device, or on a remote storage separate from thewearable extended reality appliance and the computing device. Forexample, an image similar to image 3010 as shown in FIG. 30 may begenerated and may include showing the individual's full face. In someembodiments, an AI algorithm, such as any one of the AI or machinelearning algorithms described earlier, may be used to render the imageof the individual's full face from the second perspective without thewearable extended reality appliance. For example, the AI algorithm maycombine the previously captured image of the individual's face withimage 3010 to generate the new image from the second perspective.

In some embodiments, the operations include causing the wearableextended reality appliance to present a preview of the video of theindividual interacting with the at least one virtual object from thesecond perspective while the individual is interacting with the at leastone virtual object. The preview video may be generated in a similarmanner as generating the video described earlier. For example, thegenerated video may be previewed by the user in the extended realityenvironment, such as by displaying the preview of the generated video ina separate virtual window in the extended reality environment.

In some embodiments, the operations include sharing the generated videowith at least one other individual while the individual wearing thewearable extended reality appliance is interacting with the at least onevirtual object. Generating the video may be performed in a similarmanner as described earlier. For example, the operations may includeautomatically sending the generated video to another individual. In someembodiments, the generated video may be sent to another individual byany one or more of: an email message, a social media account message, asocial media account post, a hyperlink or other type of link, or bydisplaying the generated video on a screen (e.g., a television, amonitor, a tablet, a mobile phone, a mobile device, or other displaydevice).

In some embodiments, the individual wearing the wearable extendedreality appliance may initiate the sharing of the generated video. Forexample, the individual may activate a user interface control in theextended reality environment to initiate the sharing. As anotherexample, the individual may activate a physical control on the wearableextended reality appliance or on the computing device to initiate thesharing. In some embodiments, the outside observer may initiate thesharing of the generated video. For example, the outside observer mayactivate a physical control on the wearable extended reality applianceor on the computing device to initiate the sharing.

In some embodiments, the operations include receiving input that anadditional video of the individual interacting with the at least onevirtual object from the first perspective is preferred. In someembodiments, the input may be received from the individual in theextended reality environment, for example, by the individual activatinga user interface control in the extended reality environment. In someembodiments, the input may be received from the user by the useractivating a physical control on the wearable extended reality applianceor on the computing device. In some embodiments, it may be easier toobserve how the user is interacting with the virtual object from thefirst perspective than from the second perspective. For example, if thevirtual object is a virtual screen including a text document and theuser is typing text into the document, it may be easier for the outsideobserver to read the document from the first perspective than from thesecond perspective because in the first perspective the outside observerwould see the text oriented in the same direction as the user. Asanother example, if the user is repairing a bicycle in the extendedreality environment, it would be easier for the outside observer tounderstand how the user is repairing the bicycle if the outside observerwas able to see the video from the first perspective.

In some embodiments, the operations include melding a representation ofthe at least one virtual object from the first perspective with thefirst image data to generate the additional video of the individualinteracting with the at least one virtual object from the firstperspective. The melding may be performed in a similar manner asdescribed earlier. In this embodiment, the additional video may begenerated from the first perspective in a similar manner as generatingthe video from the second perspective as described earlier. Whiledifferent image data may be used in generating the additional video(i.e., substituting the first image data for the second image data), themethods for melding and generating may be similar. For example, an imageof the user's hands in the physical environment (i.e., the first imagedata) may be melded with the virtual object from the first perspectiveto generate the additional video.

In some embodiments, the operations include switching from the videofrom the second perspective to the additional video from the firstperspective. In some embodiments, the switching may be initiated by theuser by activating a user interface control in the extended realityenvironment. In some embodiments, the switching may occur based on apredetermined action by the individual. For example, the individual mayperform a predetermined hand gesture in the extended reality environmentto activate the switching. In some embodiments, the switching may beinitiated by the user activating a physical control on the wearableextended reality appliance or on the computing device. In someembodiments, the video may begin from the second perspective and switchto being from the first perspective. In some embodiments, the video mayswitch between the first perspective and the second perspective multipletimes. In some embodiments, a “split screen” video may be generatedwhere one portion of the video (e.g., the left side) is video from thesecond perspective and another portion of the video (e.g., the rightside) is video from the first perspective.

In some embodiments, the input is received from the at least one otherindividual. For example, the other individual may be the outsideobserver. For example, the outside observer may enter a command oractivate a physical control on the computing device selectivelyconnected to the wearable extended reality appliance to initiate theswitching.

FIG. 32 is a flowchart of an exemplary method 3210 for generating videosof individuals interacting with virtual objects. The terms “generating,”“videos,” and “interacting” as used in a similar manner as describedabove. FIG. 32 is an exemplary representation of just one embodiment,and it is to be understood that some illustrated elements might beomitted and others added within the scope of this disclosure. One ormore operations of method 3210 may be performed by a processorassociated with a wearable extended reality appliance. For example, afirst processor may be located in the wearable extended realityappliance and may perform one or more operations of the method 3210. Asanother example, a second processor may be located in a computing deviceselectively connected to the wearable extended reality appliance, andthe second processor may perform one or more operations of the method3210. As another example, the first processor and the second processormay cooperate to perform one or more operations of the method 3210. Thecooperation between the first processor and the second processor mayinclude load balancing, work sharing, or other known mechanisms fordividing a workload between multiple processors.

Method 3210 may include a step 3212 of generating a presentation of anextended reality environment including at least one virtual object. Insome embodiments, the extended reality environment may be presented viathe wearable extended reality appliance. The extended realityenvironment may be generated and presented in a similar manner asdescribed above.

Method 3210 may include a step 3214 of receiving first image data fromat least a first image sensor. The terms “first image data” and “firstimage sensor” are used in a similar as described above. In someembodiments, the first image data may reflect a first perspective of anindividual wearing the wearable extended reality appliance. In someembodiments, the first image sensor may be a part of the wearableextended reality appliance.

Method 3210 may include a step 3216 of receiving second image data fromat least a second image sensor. The terms “second image data” and“second image sensor” are used in a similar as described above. In someembodiments, the second image data may reflect a second perspectivefacing the individual wearing the wearable extended reality appliance.In some embodiments, the second image sensor may be a part of acomputing device selectively connected to the wearable extended realityappliance.

Method 3210 may include a step 3218 of identifying first physical handmovements in the first image data. The terms “first physical handmovements” and “identified” are used in a similar manner as describedabove. In some embodiments, the first physical hand movements mayrepresent the individual interacting with the at least one virtualobject from the first perspective.

Method 3210 may include a step 3220 of identifying second physical handmovements in the second image data. The terms “second physical handmovements” and “identified” are used in a similar manner as describedabove. In some embodiments, the second physical hand movements mayrepresent the individual interacting with the at least one virtualobject from the second perspective.

Method 3210 may include a step 3222 of analyzing the first image data orthe second image data to determine an interaction with the at least onevirtual object. The term “analyzing” is used in a similar manner as theterm “analyzed” described above. In some embodiments, the analyzing mayinclude analyzing both the first image data and the second image data.In some embodiments, the interaction may correspond only to the firstphysical hand movements. In some embodiments, the interaction maycorrespond only to the second physical hand movements. In someembodiments, the interaction may correspond to both the first physicalhand movements and the second physical hand movements.

Method 3210 may include a step 3224 of rendering the at least onevirtual object for display from the second perspective. The term“rendering” is used in a similar manner as the term “rendered” describedabove. For example, if the user is holding a virtual puzzle cube, theuser will see a certain color combination facing in the user'sdirection. To render the puzzle cube from the second perspective (i.e.,the face of the puzzle cube that the outside observer would see), therendering may include using use artificial intelligence (AI) algorithmsmachine learning (ML) algorithms as described above and/or informationfrom the wearable extended reality device about the puzzle cube.

Method 3210 may include a step 3226 of melding the renderedrepresentation of the at least one virtual object from the secondperspective with the second image data to generate a video of theindividual interacting with the at least one virtual object from thesecond perspective. The term “melding” is used in a similar manner asthe term “rendered” described above. In some embodiments, instead of avideo, a still melded image, or a series of one or more still meldedimages may be generated.

In an alternative embodiment, the generated video may be based on onlythe second image data and the rendered virtual object from the secondperspective; i.e., the first image data may not be received or may notbe analyzed. In such alternative embodiment, the second image data fromthe second image sensor may be received. The second physical handmovements interacting with the at least one virtual object from thesecond perspective may be identified. The second image data may beanalyzed to determine an interaction with the at least one virtualobject. The at least one virtual object may be rendered from the secondperspective and may be melded with the second image data to generate thevideo. It is noted that in this alternative embodiment, the generatedvideo would be limited to the second perspective.

Some embodiments may provide a system for generating videos ofindividuals interacting with virtual objects. The system includes atleast one processor programmed to cause a wearable extended realityappliance to generate a presentation of an extended reality environmentincluding at least one virtual object; receive first image data from atleast a first image sensor, the first image data reflecting a firstperspective of an individual wearing the wearable extended realityappliance; receive second image data from at least a second imagesensor, the second image data reflecting a second perspective facing theindividual; identify in the first image data first physical handmovements interacting with the at least one virtual object from thefirst perspective; identify in the second image data second physicalhand movements interacting with the at least one virtual object from thesecond perspective; analyze at least one of the first image data or thesecond image data to determine an interaction with the at least onevirtual object; render for display a representation of the at least onevirtual object from the second perspective; and meld the renderedrepresentation of the at least one virtual object from the secondperspective with the second image data to generate a video of theindividual interacting with the at least one virtual object from thesecond perspective.

For example, the system may include system 200 shown in FIG. 2 . The atleast one processor may include processing device 360 shown in FIG. 3and/or processing device 460 shown in FIG. 4 . The steps may beperformed entirely by processing device 360, entirely by processingdevice 460, or jointly by processing device 360 and processing device.The cooperation between processing device 360 and processing device 460may include load balancing, work sharing, or other known mechanisms fordividing a workload between multiple processing devices.

Disclosed embodiments, including methods, systems, apparatuses, andnon-transitory computer-readable media, may relate to enablingcollaboration between physical writers and virtual writers, or betweenphysical writers and virtual viewers. Some embodiments involve anon-transitory computer readable medium containing instructions forcausing at least one processor to perform operations to enablecollaboration between physical writers and virtual writers. The term“non-transitory computer readable medium” may be understood as describedearlier. The term “instructions” may refer to program code instructionsthat may be executed by a computer processor. The instructions may bewritten in any type of computer programming language, such as aninterpretive language (e.g., scripting languages such as HTML andJavaScript), a procedural or functional language (e.g., C or Pascal thatmay be compiled for converting to executable code), object-orientedprogramming language (e.g., Java or Python), logical programminglanguage (e.g., Prolog or Answer Set Programming), or any otherprogramming language. In some embodiments, the instructions mayimplement methods associated with machine learning, deep learning,artificial intelligence, digital image processing, and any othercomputer processing technique. The term “processor” may be understood asdescribed earlier. For example, the at least one processor may be one ormore of server 210 of FIG. 2 , mobile communications device 206,processing device 360 of FIG. 3 , processing device 460 of FIG. 4 ,processing device 560 of FIG. 5 , and the instructions may be stored atany of memory devices 212, 311, 411, or 511, or a memory of mobilecommunications device 206.

A physical writer may be any individual. A virtual writer may be anyindividual. A virtual viewer may be any individual. Collaborationbetween physical writers and virtual writers may include interactionsbetween physical writers and virtual writers in extended realityenvironments. For example, disclosed embodiments may relate to one ormore ways for individuals working in extended reality to add annotationsto a physical surface (e.g., a physical document or whiteboard)concurrently edited in a physical space, even when the individuals arenot located in the same physical space (e.g., physical room). Disclosedembodiments may involve augmenting virtual markings of a remote writerover tangible markings of an individual wearing an extended realityappliance. Collaboration between physical writers and virtual viewersmay include interactions between physical writers and virtual viewers inextended reality environments.

Some embodiments involve receiving image data representing a hand of afirst physical writer holding a physical marking implement and engagingwith a physical surface to create tangible markings, wherein the imagedata is received from an image sensor associated with a wearableextended reality appliance worn by the first physical writer. A physicalmarking implement may include, for example, a pen, pencil, piece ofchalk, highlighter, marker, brush, or any other implement configured tocreate markings on a physical surface. The physical surface may be asurface of a physical object, which may include, for example, anotebook, whiteboard, desk, table, wall, window, touch pad, cup, mobiledevice, screen, shelf, machine, vehicle, door, chair, or any otherphysical item or object. In some embodiments, the physical surface is atleast one of a whiteboard or a paper. In some embodiments, the physicalsurface is a compilation of pages. The tangible markings may include,for example, a letter, word, sentence, paragraph, text, line, arc,freeform, shape, symbol, figure, drawing, feature, sign, or any otherindication on a physical surface.

The first physical writer may be any individual and may wear a wearableextended reality appliance. In some examples, the wearable extendedreality appliance may include an image sensor. In some examples, animage sensor separate from the wearable extended reality appliance maybe placed in the environment of the first physical writer. The imagesensor, whether part of or separate from the wearable extended realityappliance, may be configured to capture images of the scenes in front ofthe image sensor. For example, the image sensor may continuously orperiodically capture image data. The image data may represent a hand ofthe first physical writer holding a physical marking implement andengaging with a physical surface to create tangible markings. In someexamples, at least one processor associated with the wearable extendedreality appliance may receive, from the image sensor, the captured imagedata.

FIG. 33 is a schematic diagram illustrating use of an exemplary wearableextended reality appliance consistent with some embodiments of thepresent disclosure. With reference to FIG. 33 , a first physical writer3310 may be an individual. First physical writer 3310 may wear awearable extended reality appliance 3312. A hand of first physicalwriter 3310 may hold a physical marking implement 3316 and may engage(for example, via the physical marking implement 3316) with a physicalsurface 3314 to create tangible markings. The physical marking implement3316 may be, for example, a pen, a pencil, a chalk, a highlighter, amarker, a brush, or any other apparatus configured to create markings ona surface. An example of the physical surface 3314 as shown in FIG. 33may be a notebook. In some examples, the physical surface 3314 may be awhiteboard, a piece of paper, a wall, a window, a physical glasssurface, a physical table top, or a surface of any other physical objectas desired by a person of ordinary skill in the art. An image sensor,whether part of or separate from wearable extended reality appliance3312, may capture image data representing the hand of first physicalwriter 3310 holding physical marking implement 3316 and engaging withphysical surface 3314 to create tangible markings. At least oneprocessor associated with wearable extended reality appliance 3312 mayreceive the captured image data from the image sensor.

FIGS. 34, 35, 36, and 37 are schematic diagrams illustrating various usesnapshots of an example system for virtual sharing of a physical surfaceconsistent with some embodiments of the present disclosure. FIGS. 34 and37 may illustrate one or more elements as described in connection withFIG. 33 from another perspective (e.g., the perspective of firstphysical writer 3310, the perspective of wearable extended realityappliance 3312, or the perspective of the image sensor that may be partof or separate from wearable extended reality appliance 3312). Withreference to FIG. 34 , a hand of first physical writer 3310 may holdphysical marking implement 3316 and may engage with physical surface3314 to create tangible markings 3410, 3412. Examples of tangiblemarkings 3410, 3412 as shown in FIG. 34 may be two patterns of drawings.In some examples, any other type of tangible marking as desired may becreated on physical surface 3314. An image sensor associated with (e.g.,part of or separate from) wearable extended reality appliance 3312 maycapture the scenes of creating tangible markings using physical markingimplement 3316 on physical surface 3314, including, for example, thescene as shown in FIG. 34 .

Some embodiments involve transmitting information based on the imagedata to at least one computing device associated with at least onesecond virtual writer, to thereby enable the at least one second virtualwriter to view the tangible markings created by the first physicalwriter. The at least one second virtual writer may include one or moreindividuals (e.g., different from the first physical writer). In someexamples, the at least one second virtual writer may be present in oneor more locations different from a location in which the first physicalwriter may be present. For example, the at least one second virtualwriter and the first physical writer may be present in different rooms,buildings, cities, countries, or in different locations having anydesired distance therebetween. The at least one computing deviceassociated with the at least one second virtual writer may include anytype of computing device that the at least one second virtual writer mayuse (e.g., for collaborating, interacting, or communicating with thefirst physical writer and/or the wearable extended reality applianceworn by the first physical writer). In some embodiments, the at leastone computing device associated with the at least one second virtualwriter includes at least one of another wearable extended realityappliance, a desktop computer, a laptop, a tablet, or a smartphone. Insome embodiments, the at least one computing device associated with theat least one second virtual writer may include one or more computingdevices based on virtualization and/or cloud computing technologies,such as virtual machines. The at least one computing device associatedwith the at least one second virtual writer may be located in proximityto or remote from the at least one second virtual writer and may beaccessed by the at least one second virtual writer.

At least one processor associated with the wearable extended realityappliance worn by the first physical writer may process the image datacaptured by the image sensor and may, based on the processing, determineinformation for transmitting to the at least one computing deviceassociated with the at least one second virtual writer. In someexamples, the information to be transmitted to the at least onecomputing device associated with the at least one second virtual writermay represent the scenes as captured by the image sensor. For example,the transmitted information may allow the at least one second virtualwriter to view (via the at least one computing device) the scenes asviewed by the first physical writer and/or as captured by the imagesensor. The processing of the image data may include, for example,converting the captured image data into a format suitable fortransmission.

Additionally or alternatively, the information to be transmitted may bedetermined in such a manner that the information may allow one or moreaspects of the scenes captured by the image sensor to be presented tothe at least one second virtual writer. The one or more aspects mayinclude, for example, the tangible markings created by the firstphysical writer, the physical surface on which the tangible markings maybe created, and/or any other feature of the captured scenes. Forexample, the processing of the captured image data may include analyzingthe captured image data to extract features such as the tangiblemarkings created by the first physical writer and/or the physicalsurface on which the tangible markings may be created. For example, atleast one processor may analyze the image data to track the movement ofthe physical writing implement relative to the physical surface and may,based on the tracked movement, determine the created tangible markings.In some examples, at least one processor may determine the look of thephysical surface based on captured image data where the hand of thefirst physical writer and/or the physical writing implement are notcovering portions of the physical surface.

Some embodiments involve analyzing the image data received from theimage sensor to remove at least one of the hand or the physical markingimplement from the image data to thereby create modified image data, andusing the modified image data to enable the at least one second virtualwriter to view the tangible markings created by the first physicalwriter without the at least one of the hand or the physical markingimplement. For example, by deleting the representation of the at leastone of the hand or the physical marking implement and using aninpainting algorithm to fill the deleted portions, at least oneprocessor may generate the modified image data. When the image dataincludes multiple images of the physical surface (for example, multipleframes of a video of the physical surface), the deleted portions of oneimage may be filled based on pixel data corresponding to the deletedportions in other images. For example, image processing software mayrecognize hands and marking implements (e.g., using content awareprocessing), and may remove them from an image. Alternatively andequivalently, markings may be recognized and removed from an imagecontaining one or more of a hand or a marking implement. In someexamples, a generative machine learning model may be trained usingtraining examples to remove depictions of hands and/or depictions ofmarking implements from images and/or videos. An example of suchtraining example may include a sample image or video including adepiction of a sample hand and/or marking implement, together with amodified version of the sample image or video not including thedepiction of the sample hand and/or marking implement. The trainedgenerative machine learning model may be used to analyze the image datareceived from the image sensor to remove at least one of the hand or thephysical marking implement from the image data to thereby createmodified image data. In some examples, the image data may be analyzed tocalculate a convolution of at least part of the image data and therebyobtain a result value of the calculated convolution. Further, thecreation of the modified image data may be based on the result value ofthe calculated convolution. In one example, one or more pixelsassociated with a depiction of the at least one of the hand or thephysical marking implement may be identified based on the result valueof the calculated convolution. In another example, pixels values ofpixels associated with a depiction of the at least one of the hand orthe physical marking implement may be modified to new pixel values, andthe new pixel values may be determined based on the result value of thecalculated convolution.

At least one processor associated with the wearable extended realityappliance worn by the first physical writer may transmit the informationbased on the image data to the at least one computing device associatedwith the at least one second virtual writer. The at least one computingdevice associated with the at least one second virtual writer mayreceive the information from the wearable extended reality appliance andmay use the received information to cause display to the at least onesecond virtual writer, for example, of the tangible markings created bythe first physical writer. For example, the at least one computingdevice may cause display of the tangible markings, for example, via oneor more screens associated with the at least one computing device. Insome examples, the at least one computing device may include a wearableextended reality appliance and may cause display of the tangiblemarkings virtually via a display system of the wearable extended realityappliance, which may include, for example, an optical head-mounteddisplay, a monocular head-mounted display, a binocular head-mounteddisplay, a see-through head-mounted display, a helmet-mounted display,or any other type of device configured to show images to a user.

In some examples, the tangible markings may be presented to the at leastone second virtual writer with the physical surface. In some examples,the image data may be analyzed to separate the tangible markings fromthe background physical surface, and the tangible markings may bepresented to the at least one second virtual writer over a differentsurface, such as another physical surface in an environment of the atleast one second virtual writer, a virtual surface in an extendedreality environment of the at least one second virtual writer, or anyother desired surface. Similarly, the tangible markings may be presentedto the at least one second virtual viewer with the physical surface.

With reference to FIG. 34 , at least one processor associated withwearable extended reality appliance 3312 may transmit information basedon captured image data to a computing device associated with a secondvirtual writer. The transmitted information may indicate, for example,the tangible markings 3410, 3412, and/or the physical surface 3314. Insome examples, the hands of the first physical writer 3310 and/or thephysical writing implement 3316 may be not indicated in the transmittedinformation. Additionally or alternatively, the hands of the firstphysical writer 3310 and/or the physical writing implement 3316 may beindicated in the transmitted information.

With reference to FIG. 35 , an example of the computing device 3510associated with the second virtual writer that may receive theinformation transmitted from the at least one processor associated withwearable extended reality appliance 3312 is shown. Computing device 3510may include a laptop computer. FIG. 35 shows that the second virtualwriter (e.g., an individual) is using the computing device 3510 with twohands. The screen of computing device 3510 may, based on the receivedinformation, display a representation 3512 of the physical surface 3314and display representations 3514, 3516 of the tangible markings 3410,3412. The computing device 3510 may thus allow the second virtual writeror a virtual viewer to view, for example, the tangible markings 3410,3412 created by the first physical writer 3310. In some examples, one ormore visual indicators indicative of an identity of the first physicalwriter 3310 may be displayed in association with the representations3512, 3514, and/or 3516. The one or more visual indicators may include,for example, an image 3520 of the first physical writer 3310, a textualindicator 3518 of the first physical writer 3310, or any other type ofdesired indication. In some examples, the first physical writer 3310 maybe in a different location than the second virtual writer (or thevirtual viewer), and a communication channel may be established betweenthe wearable extended reality appliance 3312 and the computing device3510 (e.g., for transmission of desired information), so that the firstphysical writer 3310 and the second virtual writer (or the virtualviewer) may collaborate based on virtual sharing of a physical surfaceas described herein. In some examples, the computing device 3510 may beany other type of computing device, such as a second wearable extendedreality appliance. One or more of the elements 3512, 3514, 3516, 3518,and/or 3520 may be displayed virtually to the second virtual writerusing a display system of the second wearable extended realityappliance.

Some embodiments involve receiving from the at least one computingdevice annotation data representing additional markings in relativelocations with respect to the tangible markings created by the firstphysical writer. For example, via a user interface of the at least onecomputing device on which the tangible markings created by the firstphysical writer may be displayed to the at least one second virtualwriter, the at least one second virtual writer may input additionalmarkings in locations relative to the displayed tangible markings. Theuser interface may include, for example, an application running on alaptop computer, a virtual surface presented by a wearable extendedreality appliance, or any other interface via which the at least onesecond virtual writer may provide input. In some examples, theadditional markings may be added onto the surface on which the tangiblemarkings created by the first physical writer may be displayed by the atleast one computing device. The additional markings may include, forexample, a letter, word, sentence, paragraph, text, line, arc, freeform,shape, symbol, figure, drawing, annotation, feature, sign, or any otherindication that may be input via a computing device. The additionalmarkings may be generated using one or more of a keyboard, stylus,mouse, touch sensitive input, voice to text input, gesture command, orany other manner of adding indicia. In some examples, the additionalmarkings may be associated with one or more particular items of thetangible markings created by the first physical writer and may bedisplayed in proximity to the one or more associated items. For example,an additional marking that the at least one second virtual writer mayinput may include a comment on a particular tangible marking created bythe first physical writer. Such an additional marking may be displayedin proximity to the particular tangible marking to indicate theassociation therebetween.

Based on the input of the additional markings, the at least onecomputing device may determine the relative locations of the additionalmarkings with respect to the tangible markings created by the firstphysical writer (e.g., as the additional markings and the tangiblemarkings are on a surface presented by the at least one computingdevice). In some examples, the relative locations may be encoded basedon a coordinate system that moves with the tangible markings or with thephysical surface, and therefore the locations may be relative to thetangible markings. In some examples, the relative locations may beencoded as a distance and a relative direction with respect to at leastpart of the tangible markings.

The at least one computing device may transmit, to the wearable extendedreality appliance worn by the first physical writer, annotation datarepresenting the additional markings in relative locations with respectto the tangible markings created by the first physical writer. Theannotation data may include, for example, representations of theadditional markings and/or the relative locations of the additionalmarkings with respect to the tangible markings created by the firstphysical writer. At least one processor associated with the wearableextended reality appliance worn by the first physical writer may receivethe annotation data from the at least one computing device.

With reference to FIG. 36 , the second virtual writer may, via a userinterface of the computing device 3510, input additional markings 3610.An example of the additional markings 3610 may include a comment (“Lovethis one”) on the representation 3514 of the tangible marking 3410. Theadditional markings 3610 may include a circle around, and/or may be inproximity to, the representation 3514 of the tangible marking 3410, toindicate an association between the additional markings 3610 and therepresentation 3514 of the tangible marking 3410. The computing device3510 may transmit, to the wearable extended reality appliance 3312,annotation data representing the additional markings 3610 in relativelocations with respect to the representation 3514 of the tangiblemarking 3410. The wearable extended reality appliance 3312 may receivethe annotation data from the computing device 3510. In some examples,one or more visual indicators indicative of an identity of the secondvirtual writer may be displayed in association with the additionalmarkings 3610. The one or more visual indicators may include, forexample, an image of the second virtual writer, a textual indicator 3612of the second virtual writer, or any other type of desired indication.The computing device 3510 may additionally or alternatively transmit theone or more visual indicators to the wearable extended reality appliance3312.

Some embodiments involve in response to receiving the annotation data,causing the wearable extended reality appliance to overlay the physicalsurface with virtual markings in the relative locations. At least oneprocessor associated with the wearable extended reality appliance wornby the first physical writer may receive, from the at least onecomputing device associated with the at least one second virtual writer,the annotation data representing the additional markings in relativelocations with respect to the tangible markings created by the firstphysical writer. In response to receiving the annotation data, the atleast one processor associated with the wearable extended realityappliance may cause the wearable extended reality appliance to overlaythe physical surface with the virtual markings in the relativelocations. The virtual markings may correspond to the additionalmarkings created by the at least one second virtual writer. Theoverlaying may occur using any knowing virtual reality or extendedreality tool that causes virtual information to be displayed in aphysical environment. Overlaying the physical surface with the virtualmarkings may use a display system of the wearable extended realityappliance, which may include, for example, an optical head-mounteddisplay, a monocular head-mounted display, a binocular head-mounteddisplay, a see-through head-mounted display, a helmet-mounted display,or any other type of device configured to show images to a user. In someembodiments, the annotation data includes cursor data associated with apointing device. Some embodiments involve analyzing the cursor data todetermine the relative locations of the additional markings. Forexample, the pointing device may be of the at least one computing deviceassociated with the at least one second virtual writer. The at least onesecond virtual writer may use the pointing device to input theadditional markings and to indicate the relative locations for theadditional markings. The indicated relative locations for the additionalmarkings may be recorded in the cursor data, which may be transmitted bythe at least one computing device associated with the at least onesecond virtual writer to the wearable extended reality appliance worn bythe first physical writer. The at least one processor associated withthe wearable extended reality appliance worn by the first physicalwriter may analyze the cursor data to determine the relative locationsof the additional markings.

In some examples, causing the wearable extended reality appliance tooverlay the physical surface with the virtual markings in the relativelocations may include analyzing images captured using the image sensorassociated with the wearable extended reality appliance to determine aposition and/or an orientation of at least one of the physical surfaceor the tangible markings created by the first physical writer, anddetermining the locations to place the virtual markings based on thedetermined position and/or orientation. For example, the relativelocations of the additional markings created by the at least one secondvirtual writer with respect to the tangible markings created by thefirst physical writer as presented by the at least one computing deviceassociated with the at least one second virtual writer may be mappedonto the physical surface, so that the tangible markings and theadditional markings may be presented to the first physical writer andthe at least one second virtual writer in a similar way. This wouldenable collaboration between the first physical writer and the at leastone second virtual writer by virtual sharing of the physical surface.

With reference to FIG. 37 , at least one processor associated with thewearable extended reality appliance 3312 may receive the annotation datafrom the computing device 3510 and may, in response, cause the wearableextended reality appliance 3312 to overlay the physical surface 3314with virtual markings 3710 in the relative locations with respect to thetangible markings 3410, 3412. The virtual markings 3710 may correspondto the additional markings 3610 created by the second virtual writer.Based on the additional markings 3610 created by the second virtualwriter, the virtual markings 3710 may include a comment (“Love thisone”) on the tangible marking 3410. Based on the additional markings3610, the virtual markings 3710 may include a circle around, and/or maybe in proximity to, the tangible marking 3410, to indicate anassociation between the virtual markings 3710 and the tangible marking3410. In some examples, the wearable extended reality appliance 3312 mayreceive, from the computing device 3510, one or more visual indicatorsassociated with the additional markings 3610. The one or more visualindicators may be indicative of an identity of the second virtual writerand may be displayed virtually by the wearable extended realityappliance 3312 in association with the virtual markings 3710corresponding to the additional markings 3610. The one or more visualindicators may include, for example, an image of the second virtualwriter, a textual indicator 3712 of the second virtual writer, or anyother type of desired indication.

In some embodiments, the computing device 3510 may transmit, to thewearable extended reality appliance 3312, cursor data associated with apointing device (e.g., of the computing device 3510). For example, thesecond virtual writer may use the pointing device (which may control acursor 3522 as shown in FIGS. 35 and 36 ) to input the additionalmarkings 3610 and to indicate the relative locations for the additionalmarkings 3610. The indicated relative locations for the additionalmarkings 3610 may be recorded in the cursor data, which may betransmitted to the wearable extended reality appliance 3312. The atleast one processor associated with the wearable extended realityappliance 3312 may analyze the cursor data to determine the relativelocations of the additional markings 3610.

In some embodiments, the physical surface is a compilation of pages andthe annotation data received from the at least one computing deviceassociated with the at least one second virtual writer represents firstvirtual markings associated with a first page of the compilation andsecond virtual markings associated with a second page of thecompilation. For example, the at least one second virtual writer mayinput additional markings specific to each of one or more pages of thecompilation. Such specific markings may be different for differentpages. The at least one computing device may transmit, to the wearableextended reality appliance worn by the first physical writer, thepage-specific markings created by the at least one second virtualwriter, for presenting as page-specific virtual markings by the wearableextended reality appliance.

In some embodiments, the compilation is a notebook and the embodimentsinvolve analyzing the image data (e.g., as captured by the image sensorassociated with the wearable extended reality appliance worn by thefirst physical writer) to determine that the notebook is opened to thefirst page, and causing the wearable extended reality appliance tooverlay the first virtual markings on the first page of the notebook andexclude overlaying the second virtual markings on the first page of thenotebook. For example, at least one processor associated with thewearable extended reality appliance worn by the first physical writermay analyze specific characteristics of the pages of the notebook toidentify particular pages. In some examples, the pages of the notebookmay include page numbers, and the at least one processor may identifyparticular pages of the notebook based on the page numbers in the imagedata. In other examples, notebook pages may each contain a unique codeto enable page identification. In further examples, images of each ofone or more pages of the notebook, or other feature informationrepresenting each page, may be stored in memory, and the at least oneprocessor may compare the stored information for each page with thecaptured information for a particular page in the image data, toidentify the particular page. Based on identifying the page, the atleast one processor may cause display of virtual markings specific tothe identified page (e.g., in the annotation data), for example, byoverlaying the specific virtual markings on the identified page of thenotebook and not overlaying other virtual markings (e.g., in theannotation data) on the identified page of the notebook.

Some embodiments involve analyzing the image data to determine when thefirst page is turned, and causing the first virtual markings todisappear in response to the determination that the first page isturned. For example, at least one processor associated with the wearableextended reality appliance worn by the first physical writer may use theimage data captured by the image sensor associated with the wearableextended reality appliance to detect a gesture of the first physicalwriter turning the first page of the notebook. Additionally oralternatively, the at least one processor may analyze the image data(e.g., periodically, continuously, or when page flipping is detected) todetermine whether a currently showing page of the notebook is aparticular page (e.g., the first page), based on the content (e.g., thetangible markings and/or a page number) of the currently showing page.For example, the at least one processor may determine that the firstpage is turned based on determining that the currently showing page ofthe notebook is changing from the content of the first page to thecontent of another page. In response to the determination that the firstpage is turned, the at least one processor may cause the first virtualmarkings to disappear. For example, a machine learning model may betrained using training examples to determine when a page is turned fromimages and/or videos. An example of such training example may include asample image or a sample video, together with a label indicating whetherthe sample image or the sample video depicts a turn of a page. Thetrained machine learning model may be used to analyze the image data todetermine when the first page is turned.

Some embodiments involve analyzing the image data to determine when thefirst page is flipped back, and causing the first virtual markings toreappear in response to the determination that the first page is flippedback. For example, at least one processor associated with the wearableextended reality appliance worn by the first physical writer may use theimage data captured by the image sensor associated with the wearableextended reality appliance to detect a gesture of the first physicalwriter flipping back to the first page of the notebook. Additionally oralternatively, the at least one processor may analyze the image data(e.g., periodically or continuously) to determine whether a currentlyshowing page of the notebook is a particular page (e.g., the firstpage), based on the content (e.g., the tangible markings and/or a pagenumber) of the currently showing page. For example, the at least oneprocessor may determine that the first page is flipped back based ondetermining that the currently showing page of the notebook is changingto the content of the first page from the content of another page. Inresponse to the determination that the first page is flipped back, theat least one processor may cause the first virtual markings to reappear.In some examples, a machine learning model may be trained using trainingexamples to distinguish between different pages and recognize particularpages from images and/or videos. An example of such training example mayinclude two sample images, each image depicting a sample page, togetherwith label(s) indicating whether the two sample pages are the same pageor different pages. The trained machine learning model may be used toanalyze the image data and determine when the first page is flipped back(for example, when the first page reappears and a second pagedisappears).

Some embodiments involve causing the wearable extended reality applianceto present a virtual representation of the second page with the secondvirtual markings away from the notebook. For example, at least oneprocessor associated with the wearable extended reality appliance wornby the first physical writer may store in memory the content (e.g., thetangible markings and/or a page number) of each of one or more pages ofthe notebook, using captured image data when the notebook is showingvarious pages. When the notebook is opened to the first page (e.g., withthe first virtual markings overlaying thereon), the at least oneprocessor may cause the wearable extended reality appliance to present avirtual representation of the second page of the notebook with thesecond virtual markings (e.g., representing one or more additionalmarkings created by the at least one second virtual writer for thesecond page of the notebook).

The second virtual markings may be displayed together with the virtualrepresentation of the second page of the notebook in a similar manner asthe second virtual markings would have been displayed to overlay on thesecond page of the notebook. For example, the second virtual markingsmay be displayed on the surface of the second page in virtualrepresentation, and/or may be displayed in locations relative to thevirtual representation of the tangible markings of the second page (forexample, the relative locations may be specified by the at least onesecond virtual writer). The virtual representation of the second page ofthe notebook with the second virtual markings may be displayed away fromthe notebook. For example, the virtual representation of the second pageof the notebook with the second virtual markings may be displayed nextto, or in any other desired location relative to, the notebook (whichmay be opened to the first page), so that the first page with the firstvirtual markings overlaying thereon and the virtual representation ofthe second page of the notebook with the second virtual markings may bedisplayed at the same time to the first physical writer.

In some embodiments, the annotation data includes second image datarepresenting a hand of an additional physical writer holding a secondphysical marking implement and engaging with a second physical surfaceto create second tangible markings. Some embodiments involve analyzingthe second image data to determine the relative locations of the secondtangible markings. For example, at least one processor associated withthe wearable extended reality appliance worn by the first physicalwriter may transmit information based on the captured image data to awearable extended reality appliance worn by the additional physicalwriter, to thereby enable the additional physical writer to view thetangible markings created by the first physical writer. The wearableextended reality appliance worn by the additional physical writer may,based on receiving the information, cause display of the tangiblemarkings created by the first physical writer, for example, byoverlaying the second physical surface with the tangible markingscreated by the first physical writer. The additional physical writer mayuse a second physical marking implement to create second tangiblemarkings on the second physical surface in locations relative to thevirtual representations, of the tangible markings created by the firstphysical writer, overlaying on the second physical surface.

An image sensor associated with the wearable extended reality applianceworn by the additional physical writer may capture second image datarepresenting a hand of the additional physical writer holding the secondphysical marking implement and engaging with the second physical surfaceto create the second tangible markings. The second image data may betransmitted to the wearable extended reality appliance worn by the firstphysical writer (e.g., in the annotation data). At least one processorassociated with the wearable extended reality appliance worn by thefirst physical writer may analyze the second image data to determine therelative locations of the second tangible markings (e.g., with respectto the tangible markings created by the first physical writer) and maycause display of the second tangible markings, for example, byoverlaying the first physical surface with the second tangible markingsin the relative locations.

In some embodiments, the at least one second virtual writer includes aplurality of virtual writers. Some embodiments involve causing thewearable extended reality appliance to present in association withvirtual markings made by a virtual writer of the plurality of virtualwriters, a visual indicator indicative of an identity of the virtualwriter. For example, the identity of the virtual writer may betransmitted to the wearable extended reality appliance worn by the firstphysical writer in connection with the data indicating the virtualmarkings made by the virtual writer. The visual indicator may be, forexample, displayed in proximity to the virtual markings to indicate theassociation therebetween. In some examples, additional indications(e.g., an arrow, a linking line, or a dotted line) may be displayed toshow the association between the visual indicator and the virtualmarkings. In some examples, virtual markings made by multiple virtualwriters may be displayed by the wearable extended reality appliance wornby the first physical writer and, in association with virtual markingsmade by each of the multiple virtual writers, a visual indicatorindicative of an identity of the corresponding virtual writer may bedisplayed. In some embodiments, the visual indicator includes at leastone of an image of a virtual writer, a textual indicator of a virtualwriter, or a symbol associated with a virtual writer. The visualindicator may additionally or alternatively include any other type ofdesired indication. In some examples, the visual indicator may beconfigured by the virtual writer. For example, the virtual writer mayupload an image and enter a name for the visual indicator of the virtualwriter. With reference to FIG. 37 , one or more visual indicators (e.g.,including the textual indicator 3712) indicative of an identity of thesecond virtual writer that created the additional markings 3610 may bedisplayed virtually by the wearable extended reality appliance 3312 inassociation with the virtual markings 3710 corresponding to theadditional markings 3610.

In some embodiments, transmitting the image data to the at least onecomputing device associated with the at least one second virtual writerincludes transmitting the image data to a group of computing devicesassociated with a group of virtual writers. For example, at least oneprocessor associated with the wearable extended reality appliance wornby the first physical writer may transmit the image data to the group ofcomputing devices associated with the group of virtual writers. Someembodiments involve receiving input from the first physical writerdesignating one or more virtual writers of the group of virtual writersfor participation, and displaying virtual markings associated with theone or more virtual writers of the group of virtual writers designatedfor participation while preventing display of virtual markingsassociated with others in the group of virtual writers not designatedfor participation. For example, at least one processor associated withthe wearable extended reality appliance worn by the first physicalwriter may cause display of a listing of the group of virtual writers.The listing may allow the first physical writer to select the one ormore virtual writers for designating for participation. The listing mayinclude a menu, a list of entries, a number of tiles, or any otherdesired form. Based on receiving the first physical writer's designationof the one or more virtual writers for participation, the at least oneprocessor associated with the wearable extended reality appliance wornby the first physical writer may cause display of virtual markingsassociated with the one or more virtual writers of the group of virtualwriters designated for participation. The at least one processorassociated with the wearable extended reality appliance worn by thefirst physical writer may not cause display of virtual markingsassociated with others in the group of virtual writers, who were notdesignated for participation by the first physical writer.

In some embodiments, the at least one second virtual writer includes aplurality of virtual writers. Some embodiments involve causing thewearable extended reality appliance to display a listing of individualspermitted to view the tangible markings of the first physical writer.The listing of individuals may include, for example, one or more virtualwriters of the plurality of virtual writers. The listing of individualsmay be configured by the first physical writer. For example, the firstphysical writer may add individual(s) to, or remove individual(s) from,the listing of individuals. The listing may include a menu, a list ofentries, a number of tiles, or any other desired form. Based on thelisting of individuals, at least one processor associated with thewearable extended reality appliance worn by the first physical writermay transmit data indicating the tangible markings of the first physicalwriter to the listed individuals, so that those individuals may view thetangible markings of the first physical writer.

Some embodiments involve transmitting additional data to the at leastone computing device associated with the at least one second virtualwriter, to thereby enable the at least one second virtual writer to viewthe tangible markings with a visual indicator indicative of an identityof the first physical writer associated with the tangible markings. Forexample, at least one processor associated with the wearable extendedreality appliance worn by the first physical writer may determine theidentity of the first physical writer and may transmit additional dataindicating the identity of the first physical writer to the at least onecomputing device associated with the at least one second virtual writer.The determining of the identity of the first physical writer may bebased on, for example, the first physical writer uploading an imageand/or entering a name, symbol, or any other information associated withthe first physical writer. Based on receiving the additional data, theat least one computing device associated with the at least one secondvirtual writer may cause display of the visual indicator indicative ofthe identity of the first physical writer. The visual indicator may bedisplayed in association with (e.g., in proximity to or next to) thetangible markings of the first physical writer as displayed using the atleast one computing device associated with the at least one secondvirtual writer. In some examples, additional indications (e.g., anarrow, a linking line, or a dotted line) may be displayed to show theassociation between the visual indicator and the displayed tangiblemarkings. In some examples, the visual indicator may include at leastone of an image of the first physical writer, a textual indicator of thefirst physical writer, a symbol associated with the first physicalwriter, and/or any other type of desired indication. With reference toFIG. 35 , the computing device 3510 may receive the additional dataindicating the identity of the first physical writer 3310 from at leastone processor associated with the wearable extended reality appliance3312 and may, based on the additional data, cause display of one or morevisual indicators indicative of an identity of the first physical writer3310. The one or more visual indicators indicative of an identity of thefirst physical writer 3310 may be displayed in association with therepresentations 3512, 3514, and/or 3516, allowing the second virtualwriter to view. The one or more visual indicators may include, forexample, an image 3520 of the first physical writer 3310, a textualindicator 3518 of the first physical writer 3310, or any other type ofdesired indication. The one or more visual indicators may indicate, forexample, that the representations 3512, 3514, and/or 3516 are associatedwith (e.g., created by, used by, or belonging to) the first physicalwriter 3310.

Some embodiments involve, after overlaying the physical surface with thevirtual markings, receiving input for causing a modification in thevirtual markings. Some embodiments involve, in response to receiving theinput, causing the wearable extended reality appliance to modify thevirtual markings. For example, at least one processor associated withthe wearable extended reality appliance worn by the first physicalwriter may overlay the physical surface with the virtual markings andmay thereafter receive an input for causing the modification in thevirtual markings. The input may be received from the first physicalwriter and/or the at least one second virtual writer. For example, thefirst physical writer may provide the input via an input device of thewearable extended reality appliance. In some embodiments, the inputincludes additional image data received from the image sensor associatedwith the wearable extended reality appliance worn by the first physicalwriter. Some embodiments involve analyzing the additional image data toidentify a gesture indicting the modification in the virtual markings.The gesture may include, for example, any finger or hand motion, such asa drag, a pinch, a spread, a swipe, a tap, a pointing, a scroll, arotate, a flick, a touch, a zoom-in, a zoom-out, a thumb-up, athumb-down, a touch-and-hold, or any other action of a finger or hand.The at least one processor associated with the wearable extended realityappliance worn by the first physical writer may use image analysisalgorithms to identify the gesture directed to the virtual markings inthe additional image data. In some examples, the at least one secondvirtual writer may provide the input (e.g., via an input device or usinga gesture as captured by an image sensor) to the at least one computingdevice associated with the at least one second virtual writer, and theat least one computing device may transmit, to the wearable extendedreality appliance worn by the first physical writer, data of the input.

In some embodiments, the modification includes at least one of adeletion of at least a portion the virtual markings, changing a size ofat least a portion the virtual markings, changing a color of at least aportion the virtual markings, or changing a location of at least aportion the virtual markings. In some examples, the modification mayinclude changing a texture of at least a portion of the virtualmarkings, changing the text in the virtual markings, changing the shapeof at least a portion of the virtual markings, and/or changing anorientation of at least a portion of the virtual markings. Additionallyor alternatively, the modification may include any other type of desiredchange to at least a portion of the virtual markings.

Some embodiments involve a system for enabling collaboration betweenphysical writers and virtual writers, the system including at least oneprocessor programmed to: receive image data representing a hand of afirst physical writer holding a physical marking implement and engagingwith a physical surface to create tangible markings, wherein the imagedata is received from an image sensor associated with a wearableextended reality appliance worn by the first physical writer; transmitinformation based on the image data to at least one computing deviceassociated with at least one second virtual writer, to thereby enablethe at least one second virtual writer to view the tangible markingscreated by the first physical writer; receive from the at least onecomputing device annotation data representing additional markings inrelative locations with respect to the tangible markings created by thefirst physical writer; and in response to receiving the annotation data,cause the wearable extended reality appliance to overlay the physicalsurface with virtual markings in the relative locations.

Some embodiments involve a method for enabling collaboration betweenphysical writers and virtual writers, the method including: receivingimage data representing a hand of a first physical writer holding aphysical marking implement and engaging with a physical surface tocreate tangible markings, wherein the image data is received from animage sensor associated with a wearable extended reality appliance wornby the first physical writer; transmitting information based on theimage data to at least one computing device associated with at least onesecond virtual writer, to thereby enable the at least one second virtualwriter to view the tangible markings created by the first physicalwriter; receiving from the at least one computing device annotation datarepresenting additional markings in relative locations with respect tothe tangible markings created by the first physical writer; and inresponse to receiving the annotation data, causing the wearable extendedreality appliance to overlay the physical surface with virtual markingsin the relative locations.

FIG. 38 is a flowchart illustrating an exemplary process 3800 forvirtual sharing of a physical surface consistent with some embodimentsof the present disclosure. To the extent details of the process werepreviously discussed, all of those details may not be repeated below toavoid unnecessary repetition. With reference to FIG. 38 , in step 3810,instructions contained in a non-transitory computer-readable medium whenexecuted by a processor may cause the processor to receive image datarepresenting a hand of a first physical writer holding a physicalmarking implement and engaging with a physical surface to createtangible markings, wherein the image data may be received from an imagesensor associated with a wearable extended reality appliance(WER-Appliance) worn by the first physical writer. In step 3812,instructions contained in a non-transitory computer-readable medium whenexecuted by a processor may cause the processor to transmit informationbased on the image data to at least one computing device associated withat least one second virtual writer, to thereby enable the at least onesecond virtual writer to view the tangible markings created by the firstphysical writer. In step 3814, instructions contained in anon-transitory computer-readable medium when executed by a processor maycause the processor to receive from the at least one computing deviceannotation data representing additional markings in relative locationswith respect to the tangible markings created by the first physicalwriter. In step 3816, instructions contained in a non-transitorycomputer-readable medium when executed by a processor may cause theprocessor to, in response to receiving the annotation data, cause thewearable extended reality appliance to overlay the physical surface withvirtual markings in the relative locations.

A physical audio system is located in a physical space, for example, bymounting a speaker on a wall in a room. In this example, the speaker is“tied to” the location in the room where the speaker is mounted (forexample, a corner of the room) such that a person in the room will beable to determine where the speaker is located when she hears soundsemanating from the speaker. Similarly, a virtual audio system may betied to physical spaces. For example, a virtual speaker may be “mounted”or otherwise placed in a corner of a room in the physical environment.When a user with a wearable extended reality appliance enters the room,it may appear to the user that sounds they hear are emanating from thecorner of the room where the virtual speaker is mounted or placed.

Similar to adjusting the audio settings for a physical speaker (forexample, volume, bass, or treble), the user of the wearable extendedreality appliance in an area of the virtual audio system may change andsave audio settings (for example, volume, bass, or treble) for later usewhen the user returns to the area or when another user arrives in thelocation of the virtual audio system.

Disclosed embodiments may include methods, systems, and non-transitorycomputer readable media for facilitating tying a virtual speaker to aphysical space. It is to be understood that this disclosure is intendedto cover methods, systems, and non-transitory computer readable media,and any detail described, even if described in connection with only oneof them, is intended as a disclosure of the methods, systems, andnon-transitory computer readable media. It is noted that as used herein,the terms “physical space” and “physical environment” are understood tohave similar meanings and may be used interchangeably.

Some disclosed embodiments may be implemented via a non-transitorycomputer readable medium containing instructions for performing theoperations of a method. In some embodiments, the method may beimplemented on a system that includes at least one processor configuredto perform the operations of the method. In some embodiments, the methodmay be implemented by one or more processors associated with thewearable extended reality appliance. For example, a first processor maybe located in the wearable extended reality appliance and may performone or more operations of the method. As another example, a secondprocessor may be located in a computing device (e.g., an integratedcomputational interface device) selectively connected to the wearableextended reality appliance, and the second processor may perform one ormore operations of the method. As another example, the first processorand the second processor may cooperate to perform one or more operationsof the method. The cooperation between the first processor and thesecond processor may include load balancing, work sharing, or otherknown mechanisms for dividing a workload between multiple processors.

Some embodiments involve a non-transitory computer readable mediumcontaining instructions for causing at least one processor to performoperations to tie at least one virtual speaker to a physical space. Theterms “non-transitory computer readable medium,” “processor,” and“instructions” may be understood as described elsewhere in thisdisclosure. A “virtual speaker” refers to a location from which it maybe perceived that sounds are emitted, in an absence of a physicalspeaker at that location. For example, a user may wear headphones thatemit sound waves simulating sound as it would be heard if a physicalspeaker were located at a location where no speaker is physicallylocated. Or sound waves emitted from non-wearable physical speakers in aspace may be tuned to cause an impression that sound is being emittedfrom locations other than from the physical speakers. Tying the at leastone virtual speaker to the physical space is similar to how a physicalspeaker is tied to a physical space (for example, by mounting thespeaker in a corner of a room). For example, the virtual speaker may bevirtually “placed” in a corner of a room (i.e., the virtual speaker is“tied” to the corner of the room) or in any other location in the room.When a user with a wearable extended reality appliance enters the room,it will appear to the user that sounds are emanating from the locationin the physical space where the virtual speaker is located (e.g., thecorner of the room).

Some embodiments involve receiving, via a wireless network, a firstindication that a first wearable extended reality appliance is locatedin an area associated with a virtual speaker. The term “wearableextended reality appliance” may be understood as described elsewhere inthis disclosure. As used herein, the phrase “an area associated with avirtual speaker” is an area in a physical space from which sound isperceived to emanate in an absence of a speaker in that area. Forexample, the area may include a room in the physical space or a portionof a room in the physical space. As another example, the area mayinclude a predetermined distance (measured in any units, such ascentimeters, meters, inches, or feet) as measured from the location ofthe virtual speaker in the physical space. In some embodiments, thevirtual speaker may be an omnidirectional speaker, and the area mayinclude the predetermined distance in any direction from the virtualspeaker (e.g., a predetermined radius around the omnidirectionalspeaker). In some embodiments, the virtual speaker may be a directionalspeaker, and the area may include the predetermined distance in thedirection of the directional speaker (i.e., the predetermined distancein the direction which the directional speaker is aimed).

In some embodiments, the area of the virtual speaker may be determinedbased on a user's location. For example, a virtual speaker may bedefined a particular distance from the user at, for example, aparticular angle from a reference axis through the user. Or, the spacemay be defined by a virtual coordinate system and the speaker placed ator in an area of a particular set of coordinates (e.g., x, y or x, y,z).

In some embodiments, the first wearable extended reality appliance mayinclude Global Positioning System (GPS) and/or an indoor localizationfunctionality to determine the location of the first wearable extendedreality appliance in the physical space. The first wearable extendedreality appliance may include Wi-Fi, Bluetooth®, or other wirelesscommunication functionality as described elsewhere in this disclosurewhich may be used to determine the location of the first wearableextended reality appliance in the physical space.

In some embodiments, the physical space may include a device (e.g., aninput device, such as an integrated computational interface device asdescribed elsewhere in this disclosure) configured to receive wirelesssignals (e.g., GPS, Wi-Fi, Bluetooth®, or other wireless signal) fromthe first wearable extended reality appliance which may be used todetermine when the first wearable extended reality appliance enters thephysical space.

The first indication may include a wireless signal from the GPS, Wi-Fi,Bluetooth®, or other wireless communication functionality included inthe first wearable extended reality appliance.

Some embodiments involve transmitting to the first wearable extendedreality appliance first data corresponding to first sounds associatedwith the virtual speaker, to thereby enable a first user of the firstwearable extended reality appliance to hear the first sounds during afirst time period, wherein the first sounds correspond to first settingsof the virtual speaker. The first wearable extended reality appliancemay include (or otherwise have associated with it) headphones and/orspeakers as described elsewhere in this disclosure. The first data mayinclude data such as digital signals representing sounds in a dataformat that may be transmitted to and received by the first wearableextended reality appliance. For example, the data format may be an audiodata format such as an uncompressed format (e.g., WAV or AU), a losslesscompression format (e.g., Windows Media Audio (WMA) Lossless), or alossy compression format (e.g., WMA Lossy).

The phrase “first time period” as used herein is a time period orduration in which the first user is located in the area associated withthe virtual speaker. The first settings of the virtual speaker mayinclude any settings that may adjust the output of the virtual speaker,such as volume, bass, treble, balance (e.g., left and right balance), oran equalizer (e.g., specific frequency adjustments on a frequency bandor range).

In some embodiments, the first settings of the virtual speaker may bethe settings of the virtual speaker that exist when the first userenters the area associated with the virtual speaker. In suchembodiments, the virtual speaker settings may be associated with thevirtual speaker and not with the first user or with the first wearableextended reality appliance, meaning that the virtual speaker settingswill be the same regardless of which user or which wearable extendedreality appliance is associated with the first indication.

In some embodiments, the first settings may correspond to the firstwearable extended reality appliance and the virtual speaker may beautomatically adjusted to the first settings when the first indicationis received. The first indication may include virtual speaker settingspreferred by the first user and the settings may be stored in the firstwearable extended reality appliance. The first indication may include a“trigger” (e.g., a signal or an instruction) to the virtual speaker toretrieve settings associated with the first wearable extended realityappliance from an external storage, such as an integrated computationalinterface device or a cloud-based storage.

FIG. 39 illustrates an area in a physical space where a first user witha first wearable extended reality appliance is listening to audio atfirst settings of a virtual speaker. A physical space 3910 includes avirtual speaker 3912 tied to a location in physical space 3910. As shownin FIG. 39 , virtual speaker 3912 is located in one corner of physicalspace 3910. It is noted that the system, method, and non-transitorycomputer readable medium will perform in a similar manner if virtualspeaker 3912 is in a different location of physical space 3910, and asdescribed below, virtual speaker 3912 may be moved to differentlocations in physical space 3910. A first user 3914 wearing a firstwearable extended reality appliance 3916 is located in physical space3910 and is able to hear first sounds associated with virtual speaker3912 at first settings of virtual speaker 3912 (represented in FIG. 39by three curved lines). FIG. 39 represents the first time period asdescribed elsewhere in this disclosure, during which first user 3914listens to the first sounds at the first settings of virtual speaker3912.

Some embodiments involve receiving input associated with the firstwearable extended reality appliance during the first time period,wherein the received input is indicative of second settings for thevirtual speaker. The input may be received via a physical controllocated on an exterior portion of the first wearable extended realityappliance, such as a button, a knob, a dial, a switch, touchpad or aslider. Alternatively or additionally, the input may be received via avirtual user interface element in the extended reality environment. Forexample, the first user may activate a user interface element byinvoking a command or by a predefined hand gesture in the extendedreality environment. The input may be received as the result ofinteraction with a virtual control, such as a virtual button, knob,switch, touchpad, or slider, displayed in the user's field of view.Other examples of virtual user interface controls include a radiobutton, a checkbox, a dial, or a numerical entry field. In someembodiments, the input may be received via a voice command spoken by thefirst user. For example, the input may be received from a device pairedwith the first wearable extended reality appliance, such as the inputdevice as described elsewhere in this disclosure. For example, thedevice may include one or more physical controls, such as a button, aknob, a dial, a switch, or a slider, or may include a user interfacewith one or more user interface elements or controls. In someembodiments, the input may be received via a physical input device thatis physically separated from the first wearable extended realityappliance, such as a physical keyboard, a physical touchpad, a physicaltouchscreen, or a physical computer mouse.

The input may be a signal resulting from an environmental changedetected by a sensor associated with the wearable extended realityappliance. For example, an image sensor may detect that a meeting hasbegun, which may correspond to second desired settings for the virtualspeaker (e.g., lower volume). Similarly, the input may be a sound signalfrom a microphone that picks up increased ambient noise (e.g., from anearby construction site) corresponding to a setting of increasedvolume. Alternatively or additionally, the input may be a gesture by auser, reflecting the user's intent to adopt a second sound setting(e.g., more bass after the first user determines that a particularsoundtrack playing in the area of the virtual speaker sounds better withincreased bass).

In some embodiments, the second settings may include any settings thatmay adjust the output of the virtual speaker, similar to the firstsettings, such as volume, bass, treble, balance (e.g., left and rightbalance), or an equalizer (e.g., specific frequency adjustments on afrequency band or range). In some examples, the second settings mayinclude a new location or new orientation for the virtual speaker. Thus,the first user may be able to change any settings of the virtualspeaker.

In some embodiments, the first user may only be permitted to changecertain settings of the virtual speaker. For example, the first user mayonly be able to adjust the volume. In such embodiments, the othersettings of the virtual speaker (i.e., settings other than the volume)may be “locked” by the system or the method. For example, if the firstuser uses a virtual user interface to change the settings, only thesettings that the first user is permitted to change may be displayed.For example, if the first user is only permitted to change the volume,then only a volume control may be displayed. Controls related to othersettings may either not be displayed or may be presented in a “greyedout” manner such that while the first user may see the controls relatedto the other settings, such controls are not enabled.

Some embodiments involve transmitting to the first wearable extendedreality appliance second data corresponding to second sounds associatedwith the virtual speaker, to thereby enable the first user of the firstwearable extended reality appliance to hear the second sounds during asecond time period, wherein the second sounds correspond to the secondsettings of the virtual speaker. The second data may include data in adata format that may be transmitted to and received by the firstwearable extended reality appliance, similar to the first data asdescribed elsewhere in this disclosure. The second data may include datasuch as digital signals representing sounds in a data format that may betransmitted to and received by the first wearable extended realityappliance. In some examples, the second time period may be a time periodor duration in which the first user is located in the area associatedwith the virtual speaker and is listening to sounds at the secondsettings of the virtual speaker. In some examples, the second timeperiod may be a time period after the input indicative of secondsettings for the virtual speaker is received and/or after the first timeperiod has ended. In some implementations, the second sounds may be thesame as the first sounds, but at the second settings (e.g., at a highervolume level).

FIG. 40 illustrates the area in the physical space where the first userwith the first wearable extended reality appliance is listening to audioat second settings of the virtual speaker. A physical space 4010includes a virtual speaker 4012 tied to a location in physical space4010. As shown in FIG. 40 , virtual speaker 4012 is located in onecorner of physical space 4010. A first user 4014 wearing a firstwearable extended reality appliance 4016 is located in physical space4010 and is able to hear second sounds associated with virtual speaker4012 at second settings of virtual speaker 4012 (represented in FIG. 40by five curved lines). FIG. 40 represents the second time period asdescribed elsewhere in this disclosure, after first user 4014 haschanged the settings of virtual speaker 4012 to the second settings.

Some embodiments involve after determining that the first user and thefirst wearable extended reality appliance left the area associated withthe virtual speaker, receiving, via the wireless network, a secondindication that a second wearable extended reality appliance is locatedin the area associated with the virtual speaker. As used herein, thephrase “left the area associated with the virtual speaker” means thatthe first user and the first wearable extended reality appliance are nolonger physically present in a physical environment where the virtualspeaker is virtually located. For example, the first user may have lefta particular room, a portion of the room, or has moved more than thepredetermined distance from the location of the virtual speaker in thephysical space.

The second wearable extended reality appliance may be constructed in asimilar manner and operate in a similar way as the first wearableextended reality appliance as described elsewhere in this disclosure.The second indication may be similar to the first indication asdescribed elsewhere in this disclosure and may be used to determine whenthe second wearable extended reality appliance enters the physicalspace. The second indication may include a wireless signal from the GPS,Wi-Fi, Bluetooth®, or other wireless communication functionalityincluded in the second wearable extended reality appliance and may beused to determine the location of the second wearable extended realityappliance in the physical space.

Some embodiments involve transmitting to the second wearable extendedreality appliance third data corresponding to third sounds associatedwith the virtual speaker, to thereby enable a second user of the secondwearable extended reality appliance to hear the third sounds during athird time period, wherein the third sounds correspond to the secondsettings of the virtual speaker. The third data may include data in aformat that may be transmitted to and received by the second wearableextended reality appliance, similar to the first data as describedelsewhere in this disclosure. Additionally or alternatively, the thirddata may include data such as digital signals representing sounds in adata format that may be transmitted to and received by the secondwearable extended reality appliance. The third time period maycorrespond to a period or duration in which the second user is locatedin the area associated with the virtual speaker and is listening tosounds at the second settings of the virtual speaker. The third soundsmay be the same as the second sounds, but heard by the second user ofthe second wearable extended reality appliance. Thus, for example, afterthe first user adopts the second settings, when the second user entersthe area of the virtual speaker, the second user is exposed to thesecond sound settings.

While, in some embodiments, the second sound settings alter a soundcharacteristic, in other embodiments, the second sound setting mayadditionally or alternatively change the underlying substance of thesound. For example, the original sound may correspond to a first song,the second sound setting may be a second song, different from the firstsong. In such an example, when the second user enters the area of thevirtual speaker, the second user may be exposed to the second song, asopposed to the original first song.

FIG. 41 illustrates the area in the physical space where a second userwith a second wearable extended reality appliance is listening to audioat second settings of the virtual speaker. A physical space 4110includes a virtual speaker 4112 tied to a location in physical space4110. As shown in FIG. 41 , virtual speaker 4112 is located in onecorner of physical space 4110. A second user 4114 wearing a secondwearable extended reality appliance 4116 is located in physical space4110 and is able to hear third sounds associated with virtual speaker4112 at second settings of virtual speaker 4112 (represented in FIG. 41by five curved lines). FIG. 41 represents the third time period asdescribed elsewhere in this disclosure, during which second user 4114listens to third sounds from virtual speaker 4112 corresponding to thesecond settings.

In some embodiments, the first wearable extended reality appliance isthe second extended reality appliance. For example, the second user maybe the same person as the first user; e.g., the first user left the areaassociated with the virtual speaker and then later returned to the area.As another example, the second user may be a different person using thefirst wearable extended reality appliance; e.g., the first user and thesecond user share the same wearable extended reality appliance.

In some embodiments, the first wearable extended reality appliancediffers from the second extended reality appliance. For example, whenthe first and second users differ, the second user may enter the areaassociated with the virtual speaker wearing the second wearable extendedreality appliance. As another example, the second user may be the sameperson as the first user who may have switched from using the firstwearable extended reality appliance to using the second wearableextended reality appliance.

In some embodiments, the third data corresponding to the third sounds istransmitted to a plurality of additional wearable extended realityappliances while the first wearable extended reality appliance thatplayed the first sounds during the first time period and the secondsounds during the second time period has left the area associated withthe virtual speaker. For example, multiple users other than the firstuser may have entered the area associated with the virtual speaker afterthe first user has left the area (i.e., is absent from the area). Eachone of the plurality of additional wearable extended reality appliancesmay be associated with a different one of the multiple users in the areaand may be constructed in a similar manner and operate in a similar wayas the first wearable extended reality appliance as described elsewherein this disclosure. An indication from each of the plurality ofadditional wearable extended reality appliances may be received, toindicate that each of the plurality of additional wearable extendedreality appliances is located in the area associated with the virtualspeaker. The indication from each of the plurality of additionalwearable extended reality appliances may include a wireless signal fromthe GPS, Wi-Fi, Bluetooth®, or other wireless communicationfunctionality included in the additional wearable extended realityappliance and may be used to determine the location of the additionalwearable extended reality appliance in the physical space.

In some embodiments, the received input indicative of the secondsettings causes the second sounds and the third sounds to be played at aspecific volume level differing from an original volume level associatedwith the first settings. For example, the first user may adjust thevolume level from the original volume level of the first settings to thespecific volume level of the second settings. This adjustment may bemade separately or in addition to any one or more of the bass, treble,balance, or equalizer settings. The first user may hear the secondsounds at the specific volume level, while the second user may hear thethird sounds at the same specific volume level, meaning that the volumesettings are the same for the first user and the second user. In someembodiments, the first user may hear the second sounds at the specificvolume level, while the second user may hear the third sounds at adifferent specific volume level, meaning that the volume settings aredifferent for the first user and the second user.

In some embodiments, the received input indicative of the secondsettings causes the second sounds and the third sounds to be played at aspecific bass level differing from an original bass level associatedwith the first settings. For example, the first user may adjust the basslevel from the original bass level of the first settings to the specificbass level of the second settings. This adjustment may be madeseparately or in addition to any one or more of the volume, treble,balance, or equalizer settings.

In some embodiments, the received input indicative of the secondsettings causes the second sounds and the third sounds to be played at aspecific treble level differing from an original treble level associatedwith the first settings. For example, the first user may adjust thetreble level from the original treble level of the first settings to thespecific treble level of the second settings. This adjustment may bemade separately or in addition to any one or more of the volume, bass,balance, or equalizer settings.

In some embodiments, the received input indicative of the secondsettings causes the second sounds and the third sounds to be played atspecific settings differing from the original values associated with thefirst settings. For example, the first user may adjust the balance orthe equalizer settings from the original balance or equalizer settingsof the first settings to the specific balance or equalizer settings ofthe second settings. These adjustments may be made separately or inaddition to any one or more of the volume, bass, or treble settings.

In some embodiments, the received input indicative of the secondsettings causes the second sounds and the third sounds to play specificaudio content differing from original audio content associated with thefirst settings. For example, the first settings of the virtual speakermay remain unchanged while the specific audio content is being played.In other words, the first settings of the virtual speaker stay the sameeven if the content changes; i.e., the virtual speaker settings are nottied to the content being played.

In some embodiments, the received input indicative of the secondsettings causes the second sounds and the third sounds to play aspecific music genre differing from an original music genre associatedwith the first settings. For example, the first settings of the virtualspeaker may remain unchanged while the specific music genre is beingplayed. In other words, the first settings of the virtual speaker staythe same even if the music genre changes; i.e., the virtual speakersettings are not tied to the music genre being played.

In some embodiments, the received input indicative of the secondsettings causes the second sounds and the third sounds to play contentfrom a specific audio channel differing from an original audio channelassociated with the first settings. For example, the first settings ofthe virtual speaker may remain unchanged while the specific audiochannel is being played. In other words, the first settings of thevirtual speaker stay the same even if the audio channel changes; i.e.,the virtual speaker settings are not tied to the audio channel beingplayed. In some examples, the specific audio channel and the originalaudio channel may correspond to different radio stations. In someexamples, the specific audio channel and the original audio channel maycorrespond to different playlists. In some examples, the specific audiochannel and the original audio channel may correspond to different audiosources.

In some embodiments, the virtual speaker settings may correspond to thecontent received by the wearable extended reality appliance. In someembodiments, the virtual speaker settings may correspond to the physicalspace or to an extended reality environment associated with the wearableextended reality appliance. For example, if the physical space or theextended reality environment represents a classical music hall, thevirtual speaker settings may be adjusted to represent being in aclassical music hall with predetermined settings for bass, treble,balance, and equalizer while the user may be permitted to adjust onlythe volume. As another example, if the physical space or the extendedreality environment represents a rock concert at a large stadium, thevirtual speaker settings may be adjusted to represent being in a largestadium with predetermined settings for bass, treble, balance, andequalizer while the user may be permitted to adjust only the volume.

In some embodiments, the received input indicative of the secondsettings causes the virtual speaker to change virtual location withinthe area, the changed virtual location differs from a virtual locationassociated with the virtual speaker during the first time period, andwherein the changed virtual location is thereafter associated with thesecond time period and the third time period. For example, the settingsmay permit the first user to change the location of the virtual speakerwithin the physical space. For example, if the virtual speaker isinitially in a position next to a door in the physical space, the firstuser may change the position of the virtual speaker to be next to awindow in the physical space.

In some embodiments, the first user may be able to change the locationof the virtual speaker by selecting a location in the physical spacefrom a predetermined list of locations in the physical space. Forexample, predetermined list of locations may be provided to the firstuser via a user interface element in the extended reality environment.

In some embodiments, the first user may be able to change the locationof the virtual speaker by selecting a location in the physical spacefrom a rendering of the physical space presented in the extended realityenvironment. For example, the virtual speaker may be rendered as avirtual object in the extended reality environment and the rendering inthe extended reality environment may correspond to the area associatedwith the virtual speaker. The first user may be able to move the virtualspeaker in the extended reality environment like any other virtualobject. In some embodiments, the first user may only be able to placethe virtual speaker in predetermined locations. For example, as thefirst user moves the virtual speaker in the extended realityenvironment, the first user may be permitted to place the virtualspeaker in a predetermined location. If the first user attempts to placethe virtual speaker in a location that is not permitted, the first usermay not be able to “release” the virtual speaker to place it in thatlocation.

FIG. 42 illustrates the area in the physical space where the first userwith the first wearable extended reality appliance is listening to audioat the first settings of the virtual speaker and a location of thevirtual speaker in the area in the physical space has changed. Aphysical space 4210 includes a virtual speaker 4212 originally tied to afirst location in physical space 4210 (shown in dashed outline). FIG. 42represents the first time period as described elsewhere in thisdisclosure, after a first user 4216 has changed the location of thevirtual speaker from the first location shown by 4212 to a secondlocation for virtual speaker 4214 (shown in solid outline). As shown inFIG. 42 , the virtual speaker is moved from one corner of physical space4210 to an opposite corner of physical space 4210. The first user 4216wearing a first wearable extended reality appliance 4218 is located inphysical space 4210 and is able to hear second sounds associated withvirtual speaker 4214 at first settings of virtual speaker 4214(represented in FIG. 42 by three curved lines).

In some embodiments, in addition to moving the virtual speaker from thefirst location (indicated by reference number 4212) to the secondlocation (indicated by reference number 4214), first user 4216 may alsoadjust other settings of virtual speaker 4214, such as volume, bass,treble, balance, or an equalizer as described elsewhere in thisdisclosure.

In some embodiments, the virtual speaker is a directional virtualspeaker. A directional speaker projects sound in a specific direction(e.g., a narrow sound beam), such that a user in a path of the soundbeam can hear the sound, but other people in the physical environmentnear the user and out of the path of the sound beam cannot hear thesound. Thus, sound signals presented to a user may be tuned to simulatethe output of a directional speaker, rendering the virtual speakerdirectional. In some embodiments, the received input indicative of thesecond settings causes the virtual speaker to change orientation withrespect to a physical space, the changed orientation differs from anoriginal orientation associated with the virtual speaker during thefirst time period. The settings may permit the first user to change theorientation of the virtual speaker, by either tuning the audio output tosimulate a changed location and/or a facing direction of the virtualspeaker. For example, the first user may select the facing direction ofthe virtual speaker, which in turn tunes the audio output to simulatethe selected orientation (e.g., rotated left, right, up, down, or anycombination thereof). In some embodiments, the changed orientation ofthe virtual speaker is thereafter associated with the second time periodand the third time period. After the first user changes the orientationof the virtual speaker, the changed orientation may carry over into thesecond time period and the third time period.

FIG. 43 illustrates an area in a physical space with a directionalvirtual speaker positioned in a first orientation. A physical space 4310includes a directional virtual speaker 4312 projecting a directionalsound beam 4314 in a first orientation. It is noted that the length ofthe lines used to illustrate directional sound beam 4314 are onlyindicative of the orientation of directional sound beam 4314 and do notdefine a “length” of directional sound beam 4314 (i.e., how far a usermay be from directional virtual speaker 4312 and still be able to hearsounds from directional virtual speaker 4312). FIG. 43 represents thefirst time period as described elsewhere in this disclosure, duringwhich a first user listens to the first sounds at the first settings ofdirectional virtual speaker 4312.

FIG. 44 illustrates the area in the physical space with the directionalvirtual speaker positioned in a second orientation. A physical space4410 includes a directional virtual speaker 4412 projecting adirectional sound beam 4414 in a second orientation. It is noted thatthe length of the lines used to illustrate directional sound beam 4414are only indicative of the orientation of directional sound beam 4414and do not define a “length” of directional sound beam 4414 (i.e., howfar a user may be from directional virtual speaker 4412 and still beable to hear sounds from directional virtual speaker 4412). FIG. 44represents the second time period and the third time period as describedelsewhere in this disclosure, after the first user has changed theorientation of the directional virtual speaker and during which a firstuser or a second user listens to the second sounds or the third soundsat the second settings of directional virtual speaker 4412. In additionto changing the orientation of the directional virtual speaker from thefirst orientation (shown in FIG. 43 ) to the second orientation (shownin FIG. 44 ), the first user may also adjust other settings ofdirectional virtual speaker 4412, such as volume, bass, treble, balance,or an equalizer as described elsewhere in this disclosure.

In some embodiments, the received input indicative of the secondsettings causes a change in a size of a sound zone associated with thevirtual speaker, the changed size of the sound zone differing from anoriginal size of a sound zone associated with the virtual speaker duringthe first time period. As used herein, the term “sound zone” refers tohow far a sound can be heard from the virtual speaker. The larger thesound zone, the farther from the virtual speaker the sound can be heard.Similarly, the size of the sound zone may refer to the contours of thesound zone. Thus, the received input may tune the audio output tospecify locations in an area where sounds may or may not be heard. Insome embodiments, the sound zone may be based on a distance from thevirtual speaker's location in the physical space.

In some embodiments, the sound zone may be of any size (e.g., a width ofthe sound zone), based on characteristics of the virtual speaker. Forexample, if the virtual speaker is a directional speaker, the sound zonemay be narrow. As another example, if the virtual speaker is anomnidirectional speaker, the sound zone may be a predetermined radiusaround the virtual speaker measured in any direction, with the virtualspeaker being at a center of the sound zone. As another example, thesound zone may have a conical shape with an origin point being thelocation of the virtual speaker. In yet another example, the sound zonemay be based on a physical area, such as a room, an apartment, an officecomplex, an elevator, or a floor.

Because the speaker is a virtual speaker, the physical limitations ofvolume from a physical speaker (i.e., the farther the listener is fromthe speaker, the lower the volume appears to be to the listener) are notrelevant, meaning that the volume of the virtual speaker perceived bythe first user of the first wearable extended reality appliance may bethe same anywhere within the sound zone. For example, the first user mayset a range of X meters for the sound zone in which the virtual speakeris active. Once the first user moves outside of the range (e.g., morethan X meters), the first user will not be able to hear the sounds. Insome embodiments, at least one characteristic of sounds associated witha virtual speaker may be uniform across the sound zone. In someembodiments, at least one characteristic of sounds associated with avirtual speaker may vary across the sound zone. In some embodiments, oneor more characteristics of sounds associated with a virtual speaker maybe uniform across the sound zone while other one or more characteristicsof sounds associated with a virtual speaker may vary across the soundzone. For example, volume may be uniform across the sound zone, whilevirtual source direction may vary across the sound zone.

In some embodiments, the changed size of the sound zone is thereafterassociated with the second time period and the third time period. Afterthe first user changes the size of the sound zone, the changed soundzone size carries over into the second time period and the third timeperiod.

FIG. 45 illustrates an area in a physical space with a virtual speakerhaving a sound zone of a first size and a user with a wearable extendedreality appliance is located in the sound zone. A physical space 4510includes a virtual speaker 4512 projecting a sound zone 4514 of anoriginal size. Sound zone 4514 is shown having a conical shape with anorigin point being the location of virtual speaker 4512. Sound zone 4514may have other shapes, such as round, circular, spherical, or based onconfines of the physical space where the virtual speaker is located. Asnoted above, the virtual speaker is not subject to the same limitationsas a physical speaker, so the sound zone may be defined to have anyshape.

It is noted that the length of the dashed lines used to illustrate soundzone 4514 may indicate the original size of the sound zone, i.e., howfar a first user 4516 may be from virtual speaker 4512 and still be ableto hear sounds from virtual speaker 4512. FIG. 45 represents the firsttime period as described elsewhere in this disclosure, during which thefirst user listens to the first sounds at the first settings of virtualspeaker 4512 at the original size of the sound zone 4514.

FIG. 46 illustrates an area in a physical space with a virtual speakerhaving a sound zone of a second size and the user with the wearableextended reality appliance is located outside the sound zone. A physicalspace 4610 includes a virtual speaker 4612 projecting a sound zone 4614of a changed size. Sound zone 4614 is shown having a conical shape withan origin point being the location of virtual speaker 4612. Sound zone4614 may have other shapes, such as round, circular, spherical, or basedon confines of the physical space where the virtual speaker is located.As noted above, the virtual speaker is not subject to the samelimitations as a physical speaker, so the sound zone may be defined tohave any shape.

It is noted that the length of the dashed lines used to illustrate soundzone 4614 may indicate the changed size of the sound zone, i.e., how fara first user 4618 may be from virtual speaker 4612 and still be able tohear sounds from virtual speaker 4612. As shown in FIG. 46 , first user4618 is outside of changed size sound zone 4614, as indicated by dashedline 4616 and is thus unable to hear sounds from virtual speaker 4612.If first user 4618 moves closer to virtual speaker 4612 so that firstuser 4618 is inside the changed size sound zone 4614 (i.e., on the sideof dashed line 4616 closer to virtual speaker 4612), then first user4618 would be able to hear the sounds from virtual speaker 4612.

FIG. 46 represents the second time period and the third time period asdescribed elsewhere in this disclosure, after the first user has changedthe size of the sound zone of the virtual speaker and during which afirst user or a second user listens to the second sounds or the thirdsounds at the second settings of virtual speaker 4612. In someembodiments, in addition to changing the size of the sound zone ofvirtual speaker from the original size (shown in FIG. 45 ) to thechanged size (shown in FIG. 46 ), the first user may also adjust othersettings of virtual speaker 4612, such as volume, bass, treble, balance,or an equalizer as described elsewhere in this disclosure.

Some embodiments involve receiving an additional indication during thefirst time period that an additional wearable extended reality applianceis located in the area associated with the virtual speaker. Theadditional indication may be received in a similar manner as the firstindication and may include similar information as the first indicationas described elsewhere in this disclosure. Additionally oralternatively, the additional indication may include a wireless signalfrom the GPS, Wi-Fi, Bluetooth®, or other wireless communicationfunctionality included in the additional wearable extended realityappliance to determine the location of the additional wearable extendedreality appliance in the physical space. In some embodiments, theadditional wearable extended reality appliance may be constructed in asimilar manner and operate in a similar way as the first wearableextended reality appliance as described elsewhere in this disclosure.

Some embodiments involve transmitting to the additional wearableextended reality appliance fourth data associated with the virtualspeaker to thereby enable the additional wearable extended realityappliance to present sounds corresponding to the first settings duringthe first time period and the second time period. The fourth data mayinclude data in a format that may be transmitted to and received by theadditional wearable extended reality appliance, similar to the firstdata as described elsewhere in this disclosure. The first settings arethe same settings of the virtual speaker as used in connection with thefirst sounds (e.g., the same settings for volume, bass, treble, balance,and equalizer). The user of the additional wearable extended realityappliance may hear fourth sounds corresponding to the fourth data at thefirst settings of the virtual speaker, for example, at the originalvolume level of the first settings. Some other embodiments involvetransmitting to the additional wearable extended reality appliancefourth data associated with the virtual speaker to thereby enable theadditional wearable extended reality appliance to present soundscorresponding to the first settings during the first time period, andtransmitting to the additional wearable extended reality appliance fifthdata associated with the virtual speaker to thereby enable theadditional wearable extended reality appliance to present soundscorresponding to the second settings during the second time period.

In some embodiments, an additional user (with the additional wearableextended reality appliance) may be in the same physical space as thefirst user. For example, the first user and the additional user may belistening to the same content at the first settings of the virtualspeaker (i.e., the fourth data may be the same as the first data).Alternatively, the first user and the additional user may be listeningto different content at the first settings of the virtual speaker (i.e.,the fourth data may be different from the first data).

FIG. 47 illustrates an area in a physical space where a first user witha first wearable extended reality appliance and a second user with asecond wearable extended reality appliance are listening to audio atfirst settings of a virtual speaker. A physical space 4710 includes avirtual speaker 4712 tied to a location in physical space 4710. A firstuser 4714 wearing a first wearable extended reality appliance 4716 islocated in physical space 4710 and is able to hear first soundsassociated with virtual speaker 4712 at first settings of virtualspeaker 4712 (represented in FIG. 47 by three curved lines). A seconduser 4718 wearing an additional wearable extended reality appliance 4720is located in physical space 4710 and is able to hear fourth soundsassociated with virtual speaker 4712 at first settings of virtualspeaker 4712 (represented in FIG. 47 by three curved lines). As shown inFIG. 47 , both first user 4714 and second user 4718 may hear sounds fromvirtual speaker 4712 (first sounds and fourth sounds, respectively) atthe first settings of virtual speaker 4712.

Some embodiments involve receiving an additional indication during thefirst time period that an additional wearable extended reality applianceis located in the area associated with the virtual speaker. Theadditional indication may be received in a similar manner as the firstindication and may include similar information as the first indicationas described elsewhere in this disclosure. For example, the additionalindication may include a signal that may be used by a device (e.g.,input device, integrated computational interface device, or computingdevice) to determine that the additional wearable extended realityappliance is located in the area associated with the virtual speaker.The additional wearable extended reality appliance may be constructed ina similar manner and operate in a similar way as the first wearableextended reality appliance as described elsewhere in this disclosure.

Some embodiments involve transmitting to the additional wearableextended reality appliance fourth data associated with the virtualspeaker to thereby enable the additional wearable extended realityappliance to present sounds corresponding to the second settings duringthe second time period. The fourth data may include data in a formatthat may be transmitted to and received by the additional wearableextended reality appliance, similar to the first data as describedelsewhere in this disclosure. The fourth data may include data such asdigital signals representing sounds in a data format that may betransmitted to and received by the additional wearable extended realityappliance.

In some embodiments, an additional user (with the additional wearableextended reality appliance) may be in the same physical space as thefirst user. For example, the first user and the additional user may belistening to the same content at the second settings of the virtualspeaker (i.e., the fourth data may be the same as the first data).Alternatively, the first user and the additional user may be listeningto different content at the second settings of the virtual speaker(i.e., the fourth data may be different from the first data).

Some embodiments involve obtaining information associated with thesecond indication. For example, the information associated with thesecond indication may include information about the second wearableextended reality appliance, such as a device identifier, or informationabout the second user of the second wearable extended realityapplication, such as a user identifier. For example, the deviceidentifier and the user identifier may include an alphanumeric stringincluding an identifier code associated with the second wearableextended reality appliance and/or the second user. In other examples,the information associated with the second indication may includeinformation about the second wearable extended reality appliance, suchas a type, a settings, a location, an orientation, an activityindicator, and so forth.

Some embodiments involve accessing a plurality of sound playing rulesdefining virtual speaker settings, based on the obtained information.The plurality of sound playing rules may be stored on the wearableextended reality appliance or may be stored on a device separate fromthe wearable extended reality appliance, for example, on an input deviceor on a cloud-based server in communication with the wearable extendedreality appliance. The plurality of sound playing rules may be accessedby the wearable extended reality appliance. In some embodiments, theplurality of sound playing rules may be accessed by a device located inthe physical space and in communication with the wearable extendedreality appliance.

In some embodiments, one or more of the plurality of sound playing rulesmay be associated with a device identifier or a user identifier. Forexample, a parent may create one or more sound playing rules for a childand associate those sound playing rules with a user identifiercorresponding to the child such that when the child wears the wearableextended reality appliance, the sound playing rules associated with thechild are implemented. For example, the sound playing rule for the childmay be that the child is not able to adjust the settings of the virtualspeaker or that the child may only raise the volume setting to apredetermined maximum (which may be a lower value than a maximumpossible volume setting). As another example, the sound playing rule forthe child may be that the child is limited to playing certain types ofcontent (e.g., age-appropriate content as determined by the rulecreator). As used herein, the term “implemented” in connection with thesound playing rules means that the sound playing rules are used by thewearable extended reality appliance to control the settings associatedwith the virtual speaker. In some embodiments, one or more of theplurality of sound playing rules may be associated with an activityassociated with the second wearable extended reality appliance. Forexample, for some activities, the sound playing rules may include amaximum volume. In some embodiments, one or more of the plurality ofsound playing rules may be associated with a type of the second wearableextended reality appliance, for example due to hardware limitations ofthe second wearable extended reality appliance.

Some embodiments involve determining that an existence of the secondwearable extended reality appliance in the area associated with thevirtual speaker corresponds to a specific sound playing rule. Theexistence of a second wearable extended reality appliance in the areaassociated with a virtual speaker may be determined by signals sent bythat appliance to either the first appliance or to a central controller.Those signals may be based on a location system such as GPS, Wi-Fi, orBluetooth® related location systems. Additionally or alternatively, theexistence may be determined by image data captured by either the secondwearable extended reality appliance or another wearable extended realityappliance. Based on the determination of the existence of the secondwearable extended reality appliance in a particular area, acorrespondence to a specific sound playing rule may be determined. Forexample, the second indication may indicate that the second wearableextended reality appliance is in the area associated with the virtualspeaker. As described elsewhere in this disclosure, the secondindication may also include an identifier associated with the secondwearable extended reality appliance. In some embodiments, the secondindication may include a wireless signal from the GPS, Wi-Fi,Bluetooth®, or other wireless communication functionality included inthe second wearable extended reality appliance and may be used todetermine the location of the second wearable extended reality appliancein the physical space. In some embodiments, one or more sound playingrules associated with the second wearable extended reality appliance maybe determined based on the second indication. For example, if thephysical space is a museum and the audio is associated with an exhibitin the physical space, the sound playing rule may be that only thevolume setting may be changed by a user and when the second wearableextended reality appliance enters the area associated with the virtualspeaker, the volume setting may be adjusted to an initial value (e.g.,first settings) which may be the same for all users. Then, once the useris in the physical space, the user may adjust the volume setting.

Some embodiments involve implementing the specific sound playing rule todetermine actual settings of the third sounds heard by the second userduring the third time period. The specific sound playing rule mayinclude setting information for the virtual speaker, and implementingthe specific sound playing rule may change the virtual speaker settingsto match the settings indicated by the specific sound playing rule. Thespecific sound playing rule may be implemented by the second wearableextended reality appliance by controlling the settings (e.g., the secondsettings) associated with the virtual speaker. An input deviceassociated with the second wearable extended reality appliance mayimplement the specific sound playing rule.

In some embodiments, the specific sound playing rule is that a specificsetting change only affects devices that initiate the specific settingchange. In this situation, when a specific sound playing rule is set viaa particular device, the rule may only impact the device through whichthe rule was set. For example, if the specific sound playing rule isassociated with a particular wearable extended reality appliance (e.g.,by association via an identifier of the wearable extended realityappliance), then the change in the virtual speaker settings may onlyapply to that particular wearable extended reality appliance.

In some embodiments, the obtained information associated with the secondindication includes information about a time of day. The time of day maybe a current time in a time zone where the wearable extended realityappliance is located.

Some embodiments involve determining whether the condition of thespecific sound playing rule is met based on the information about thetime of day. If the specific sound playing rule includes a condition,the condition may be based on a current time of day. For example, acondition might be that before 5 pm, a virtual speaker plays hard rockmusic, after 5 pm, the music genre switches to light rock, and after 8pm, mood music kicks-in. By way of another example, if the wearableextended reality appliance includes a physical speaker, the specificsound playing rule may limit the volume of the physical speaker if it islater than 11:00 PM. In some embodiments, the time of day condition isone of two or more conditions of the specific sound playing rule. Forexample, the sound playing rule may include a time of day condition anda predetermined distance condition (e.g., that the wearable extendedreality appliance be located within a predetermined distance of thevirtual speaker).

Some embodiments include transmitting to the second wearable extendedreality appliance the third data corresponding to the third sounds thatcorrespond to the second settings of the virtual speaker when acondition of the specific content sound playing rule is met. The soundplaying rule may include a condition and the sound playing rule may beimplemented (i.e., the third sounds are played using the second settingsof the virtual speaker) if the condition is met. For example, acondition may include that the user is located within a predetermineddistance (measured in any unit of measurement, such as meters,centimeters, feet, or inches) of the location of the virtual speaker inthe physical space. If, for example, the physical space is a museum withmultiple exhibits and each exhibit has its own virtual speakerassociated with it, the sound playing rule may be that the user of thewearable extended reality appliance needs to be within a predetermineddistance of the virtual speaker before the audio associated with theexhibit begins playing. In such circumstances, the user would need to berelatively close to the exhibit (within the predetermined distance)before the associated audio started playing, to avoid user confusion byhaving audio associated with a different exhibit playing.

Some embodiments include transmitting to the second wearable extendedreality appliance fourth data corresponding to fourth sounds thatcorrespond to the first settings of the virtual speaker when thecondition of the specific sound playing rule is unmet. The sound playingrule may include a condition and the sound playing rule may be notimplemented (i.e., the fourth sounds are played using the first settingsof the virtual speaker) if the condition is not met. Continuing theexample of when the physical space is a museum, if the user walksbetween different rooms in the museum, the sound playing rule may bethat music (i.e., fourth sounds) is played while the user is walking andcontinues to play until the user is located within a sound zone of avirtual speaker associated with an exhibit in the museum. In thisexample, the sound playing rule may be associated with the exhibit theuser is walking toward and the condition may be that the user be withina predetermined distance of the exhibit (i.e., the condition is met)before the audio associated with the exhibit starts playing, otherwisemusic is played (i.e., the condition is not met).

In some embodiments, audio data captured using at least one audio sensorincluded in the second wearable extended reality appliance may beanalyzed to detect sounds in the physical environment of the secondwearable extended reality appliance (such as ambient noise, a personspeaking, music, and so forth). Further, in response to the detection ofthe sounds in the physical environment of the second wearable extendedreality appliance, actual settings of the third sounds heard by thesecond user during the third time period may be determined, for example,based on characteristics of the sounds in the physical environment ofthe second wearable extended reality appliance. For example, in responseto high ambient noise levels, a volume of the third sounds may beincreased. In another example, in response to a detection of a personspeaking in the physical environment of the second wearable extendedreality appliance, a volume of the third sounds may be decreased. Insome examples, the audio data may be analyzed using a voice recognitionalgorithm to determine whether the person speaking in the physicalenvironment of the second wearable extended reality appliance is thesecond user. When the person speaking is the second user, the volume ofthe third sounds may be decreased, while when the person speaking is notthe second user, the volume of the third sounds may be increased, may beunmodified, or may be decreased less than when the person speaking isthe second user. In some examples, the audio data may be analyzed todetermine whether the person speaking in the physical environment of thesecond wearable extended reality appliance is speaking to the seconduser. When the person is speaking to the second user, the volume of thethird sounds may be decreased, while when the person is not speaking tothe second user, the volume of the third sounds may be increased, may beunmodified, or may be decreased less than when the person is speaking tothe second user. For example, a machine learning model (such as aclassification model) may be trained using training examples todetermine from audio whether people are speaking to the second user. Anexample of such training example may include a sample audio, togetherwith a label indicating whether the sample audio includes speechdirected to the second user. The trained machine learning model may beused to analyze the audio data captured using at least one audio sensorincluded in the second wearable extended reality appliance and determinewhether the person speaking in the physical environment of the secondwearable extended reality appliance is speaking to the second user.

In some embodiments, additional data associated with other sounds forpresentation to the second user during the third time period may bereceived, the other sounds may not be associated with the virtualspeaker. For example, the other sounds may be associated with adifferent virtual speaker, may be associated with an app, may beassociated with a virtual object that is not a virtual speaker, and soforth. Further, the additional data may be analyzed to determine actualsettings of the third sounds heard by the second user during the thirdtime period. For example, the determination of the actual settings maybe based on a category associated with the other sounds, may be based ona volume associated with the other sounds, may be based on a position ofa virtual source associated with the other sounds, and so forth.

FIG. 48 is a flowchart of an exemplary method 4810 for tying a virtualspeaker to a physical space. FIG. 48 is an exemplary representation ofjust one embodiment, and it is to be understood that some illustratedelements might be omitted and others added within the scope of thisdisclosure. One or more operations of method 4810 may be performed by aprocessor associated with a wearable extended reality appliance. Forexample, a first processor may be located in the wearable extendedreality appliance and may perform one or more operations of the method4810. As another example, a second processor may be located in acomputing device selectively connected to the wearable extended realityappliance, and the second processor may perform one or more operationsof the method 4810. As another example, the first processor and thesecond processor may cooperate to perform one or more operations of themethod 4810. The cooperation between the first processor and the secondprocessor may include load balancing, work sharing, or other knownmechanisms for dividing a workload between multiple processors.

Method 4810 includes a step 4812 of receiving, for example via awireless network, a first indication that a first wearable extendedreality appliance is located in an area associated with a virtualspeaker. The terms “first indication,” “first wearable extended realityappliance,” “virtual speaker,” and “area associated with the virtualspeaker” have a similar meaning as described elsewhere in thisdisclosure.

In some embodiments, the physical space may include a device configuredto receive wireless signals (e.g., an input device, such as anintegrated computational interface device as described elsewhere in thisdisclosure) from the first wearable extended reality appliance which maybe used to determine when the first wearable extended reality applianceenters the physical space. The first indication may include a wirelesssignal from GPS, Wi-Ft, Bluetooth®, or other wireless communicationfunctionality included in the first wearable extended reality appliance.

Method 4810 includes a step 4814 of transmitting to the first wearableextended reality appliance first data corresponding to first soundsassociated with the virtual speaker, to thereby enable a first user ofthe first wearable extended reality appliance to hear the first soundsduring a first time period, wherein the first sounds correspond to firstsettings of the virtual speaker. In some embodiments, the first wearableextended reality appliance may include headphones and/or speakers asdescribed elsewhere in this disclosure. The terms “first data,” “firstsounds,” and “first time period” have a similar meaning as describedelsewhere in this disclosure.

Method 4810 includes a step 4816 of receiving input associated with thefirst wearable extended reality appliance during the first time period,wherein the received input is indicative of second settings for thevirtual speaker. The terms “input” and “second settings” have a similarmeaning as described elsewhere in this disclosure. In some embodiments,the input may be received in a manner similar as described elsewhere inthis disclosure.

Method 4810 includes a step 4818 of transmitting to the first wearableextended reality appliance second data corresponding to second soundsassociated with the virtual speaker, to thereby enable the first user ofthe first wearable extended reality appliance to hear the second soundsduring a second time period, wherein the second sounds correspond to thesecond settings of the virtual speaker. The terms “second data,” “secondsounds,” and “second time period” have a similar meaning as describedelsewhere in this disclosure.

Method 4810 includes a step 4820 of after determining that the firstuser and the first wearable extended reality appliance left the areaassociated with the virtual speaker, receiving via the wireless network,a second indication that a second wearable extended reality appliance islocated in the area associated with the virtual speaker. The terms “leftthe area associated with the virtual speaker,” “second indication,” and“second wearable reality appliance” have a similar meaning as describedelsewhere in this disclosure.

Method 4810 includes a step 4822 of transmitting to the second wearableextended reality appliance third data corresponding to third soundsassociated with the virtual speaker, to thereby enable a second user ofthe second wearable extended reality appliance to hear the third soundsduring a third time period, wherein the third sounds correspond to thesecond settings of the virtual speaker. The terms “third data,” “thirdsounds,” and “third time period” have a similar meaning as describedelsewhere in this disclosure.

Some embodiments provide a system for tying at least one virtual speakerto a physical space. The system includes at least one processorprogrammed to receive, via a wireless network, a first indication that afirst wearable extended reality appliance is located in an areaassociated with a virtual speaker; transmit to the first wearableextended reality appliance first data corresponding to first soundsassociated with the virtual speaker, to thereby enable a first user ofthe first wearable extended reality appliance to hear the first soundsduring a first time period, wherein the first sounds correspond to firstsettings of the virtual speaker; receive input associated with the firstwearable extended reality appliance during the first time period,wherein the received input is indicative of second settings for thevirtual speaker; transmit to the first wearable extended realityappliance second data corresponding to second sounds associated with thevirtual speaker, to thereby enable the first user of the first wearableextended reality appliance to hear the second sounds during a secondtime period, wherein the second sounds correspond to the second settingsof the virtual speaker; after determining that the first user and thefirst wearable extended reality appliance left the area associated withthe virtual speaker, receive via the wireless network, a secondindication that a second wearable extended reality appliance is locatedin the area associated with the virtual speaker; and transmit to thesecond wearable extended reality appliance third data corresponding tothird sounds associated with the virtual speaker, to thereby enable asecond user of the second wearable extended reality appliance to hearthe third sounds during a third time period, wherein the third soundscorrespond to the second settings of the virtual speaker.

For example, the system may include system 200 shown in FIG. 2 . The atleast one processor may include processing device 360 shown in FIG. 3and/or processing device 460 shown in FIG. 4 . The steps may beperformed entirely by processing device 360, entirely by processingdevice 460, or jointly by processing device 360 and processing device.The cooperation between processing device 360 and processing device 460may include load balancing, work sharing, or other known mechanisms fordividing a workload between multiple processing devices.

When using a wearable extended reality appliance to view and/or interactwith virtual objects, some virtual objects may be in a field of viewwhile some other virtual objects may be outside the field of view, asexplained in more detail below. As the field of view changes (forexample, due to head movements of the user or due to changes to displayparameters) or the positions of the virtual objects change, virtualobjects may enter and/or exit the field of view. Virtual objects maychange due to many different triggers, as described in more detailbelow. When a virtual object changes while the virtual object is outsidethe field of view, it may take a while before the user looks to thedirection of the virtual object and notices the change. Therefore, it isdesired to provide a notification indicative of the change noticeable bythe user even when the virtual object is outside the field of view.However, when the virtual object is in the field of view, the change isimmediately and directly noticeable, and providing a supplementarynotification indicative of the change may cause clutter or undesiredabundance of notifications. Disclosed embodiments, including methods,systems, apparatuses, and non-transitory computer-readable media, mayrelate to initiating location-driven sensory prompts reflecting changesto virtual objects. Some embodiments involve a non-transitory computerreadable medium containing instructions for performing operationsconfigured to initiate location-driven sensory prompts reflectingchanges to virtual objects. The term “non-transitory computer readablemedium” may be understood as described earlier. The term “instructions”may refer to program code instructions that may be executed by acomputer processor. The instructions may be written in any type ofcomputer programming language, such as an interpretive language (e.g.,scripting languages such as HTML and JavaScript), a procedural orfunctional language (e.g., C or Pascal that may be compiled forconverting to executable code), object-oriented programming language(e.g., Java or Python), logical programming language (e.g., Prolog orAnswer Set Programming), or any other programming language. In someembodiments, the instructions may implement methods associated withmachine learning, deep learning, artificial intelligence, digital imageprocessing, and any other computer processing technique. The term“processor” may be understood as described earlier. For example, the atleast one processor may be one or more of server 210 of FIG. 2 , mobilecommunications device 206, processing device 360 of FIG. 3 , processingdevice 460 of FIG. 4 , processing device 560 of FIG. 5 , and theinstructions may be stored at any of memory devices 212, 311, 411, or511, or a memory of mobile communications device 206.

A virtual object may refer to a visual representation rendered by acomputing device (e.g., a wearable extended reality appliance) andconfigured to represent an object. A virtual object may include, forexample, an inanimate virtual object, an animate virtual object, virtualfurniture, a virtual decorative object, a virtual widget, a virtualscreen, or any other type of virtual representation of any object orfeature. In some examples, a virtual object may be associated with acommunications application, a news application, a gaming application, atiming application, a word-processing application, a data-processingapplication, a presentation application, a reading application, abrowsing application, a messaging application, or any other type ofapplication. A change to a virtual object may include any type ofmodification, alteration, variation, adjustment, rearrangement,reordering, adaptation, reconstruction, transformation, or revision tothe virtual object. The change to the virtual object may include achange to any aspect of the virtual object, including the appearance ofthe virtual object, the associated content of the virtual object, theassociated functions of the virtual object, the status of the virtualobject, the state of the virtual object, the associated data of thevirtual object, or any other feature of the virtual object. In someembodiments, the change to the virtual object may include, for example,an incoming message, a received notification, a news update, anoccurrence of an event, a request for user action, a receivedadvertisement, a trigger for displaying a user interface, or any otherupdate associated with the virtual object. In some examples, the changeto the virtual object may include any action, function, and/or datadirected towards the virtual object.

A sensory prompt may refer to any indication that may be configured tobe sensed by an individual. A sensory prompt may relate to any sense ofan individual, such as sight, smell, touch, taste, hearing, or any otherability of an individual to gather information. In some examples, asensory prompt may be used to provide a notification to a user. Forexample, a sensory prompt may include a visual notification, an audiblenotification, or a tactile notification. A computing device may cause asensory prompt to be generated (e.g., via one or more output devices,such as a screen, a speaker, or a vibrator), based on one or moretriggering events, such as a change to a virtual object rendered by thecomputing device.

Initiating location-driven sensory prompts reflecting changes to virtualobjects may include causing a sensory prompt reflecting a change to avirtual object based on a location associated with the virtual object.For example, a change to a virtual object may trigger different sensoryprompts if the virtual object is located in different locations. Thedifferent locations of the virtual object that may cause differentsensory prompts may be, for example, differentiated based on a field ofview of a wearable extended reality appliance that may cause display ofthe virtual object. Disclosed embodiments may include, for example,detecting when a change happens to a virtual object outside a currentfield of view of the wearable extended reality appliance. Based onidentifying the change, the wearable extended reality appliance mayprovide a sensory prompt to notify and/or inform the user of the changeto the virtual object. The sensory prompt may be different from asensory prompt that may be triggered by a change to the virtual objectif the virtual object is within the field of view of the wearableextended reality appliance.

Some embodiments involve enabling interaction with a virtual objectlocated in an extended reality environment associated with a wearableextended reality appliance. The term “extended reality environment” maybe understood as described earlier. The virtual object (also describedearlier) may be located in any desired location in the extended realityenvironment. For example, the virtual object may be displayed (e.g., viathe wearable extended reality appliance) as being placed on a physicalwall, as being placed on a virtual wall rendered by the wearableextended reality appliance, as being placed on a virtual whiteboardrendered by the wearable extended reality appliance, as being placed(e.g., floating) in a space without being connected to other objects(either physical or virtual) in the space, or in any other desiredlocation.

Interaction with the virtual object may refer to any action from a userto the virtual object or from the virtual object to a user. Interactionwith the virtual object may include, for example, any action of a userthat may interface the virtual object, such as an instruction input by auser to the virtual object, a command provided by a user to the virtualobject, a gesture of a user directed to the virtual object, or any otherinput that may be provided by a user to the virtual object. The actionof a user that may interface with the virtual object may be via an inputdevice of a wearable extended reality appliance that may cause displayof the virtual object. Additionally or alternatively, interaction withthe virtual object may include any action of the virtual object that mayinterface with a user, such as an output image of the virtual object fora user, an output sound of the virtual object for a user, output text ofthe virtual object for a user, or any other output that may be providedby the virtual object to a user. The action of the virtual object thatmay interface a user may be via an output device of a wearable extendedreality appliance that may cause display of the virtual object.

At least one processor associated with a wearable extended realityappliance may, for example, cause display of a virtual object located inan extended reality environment associated with the wearable extendedreality appliance and enable interaction with the virtual object locatedin the extended reality environment associated with the wearableextended reality appliance. For example, the at least one processor mayactivate input devices of the wearable extended reality appliance and/oroutput devices of the wearable extended reality appliance, for a user tointeract with the virtual object. Additionally or alternatively, the atleast one processor may configure parameters, settings, functions,and/or instructions of a system of the wearable extended realityappliance, to allow a user to interact with the virtual object.

FIGS. 49, 50, 51, and 52 are schematic diagrams illustrating various usesnapshots of an example system for initiating sensory prompts forchanges based on a field of view consistent with some embodiments of thepresent disclosure. With reference to FIG. 49 , a user 4910 may wear awearable extended reality appliance 4912. The wearable extended realityappliance 4912 may provide an extended reality environment 4914 to theuser 4910. The wearable extended reality appliance 4912 may causedisplay of one or more virtual objects 4918, 4920, 4922 in the extendedreality environment 4914. An example of the virtual object 4918 may be avirtual screen. An example of the virtual object 4920 may be an icon orwidget for an email application. An example of the virtual object 4922may be an icon or widget for a clock application. At least one processorassociated with the wearable extended reality appliance 4912 may enablethe user 4910 to interact with the virtual objects 4918, 4920, 4922located in the extended reality environment 4914.

Some embodiments involve receiving data reflecting a change associatedwith the virtual object. A change associated with a virtual object mayinclude any type of modification, alteration, variation, adjustment,rearrangement, reordering, adaptation, reconstruction, transformation,or revision associated with the virtual object. The change associatedwith the virtual object may include a change to any aspect of thevirtual object, including the appearance of the virtual object, theassociated content of the virtual object, the associated functions ofthe virtual object, the status of the virtual object, the state of thevirtual object, the associated data of the virtual object, or any otherfeature of the virtual object. In some embodiments, the changeassociated with the virtual object may include, for example, an incomingmessage, a received notification, a news update, an occurrence of anevent, a request for user action, a received advertisement, a triggerfor displaying a user interface, or any other update associated with thevirtual object. In some examples, the change associated with the virtualobject may include any action, function, and/or data directed towardsthe virtual object.

At least one processor associated with the wearable extended realityappliance may, for example, receive data reflecting the changeassociated with the virtual object. In some examples, the data may bereceived from another computing device. In some examples, the data maybe received from an application, a function, and/or any other entityrunning on the wearable extended reality appliance. In some examples,the data may be received from an application, a function, and/or anyother entity associated with the virtual object. In some examples, thedata may be generated by and/or received from the virtual object.Additionally or alternatively, the data may be received from any otherdesired entity.

In some embodiments, the virtual object is associated with acommunications application and the change associated with the virtualobject involves at least one of an incoming message or a receivednotification. A communications application may include, for example, anemail application, a social network application, an instant messagingapplication, a phone application, or any other application that may beconfigured to communicate with another entity. The virtual object mayinclude an icon, a widget, a symbol, a window, or any other userinterface for the communications application. At least one processorassociated with the wearable extended reality appliance may implementthe communications application, and may receive (e.g., from anothercomputing device or any other desired entity) data indicating anincoming message for the communications application and/or anotification for the communications application. The incoming messageand/or the notification may include, for example, an email, a post, atext message, a phone call or voice message, or any other informationthat may be received by the communications application.

In some embodiments, the virtual object is associated with a newsapplication and the change associated with the virtual object involves anews update. A news application may refer to any application that may beconfigured to provide information of current events via one or more ofvarious media (e.g., broadcasting or electronic communication). Thevirtual object may include an icon, a widget, a symbol, a window, or anyother user interface for the news application. At least one processorassociated with the wearable extended reality appliance may implementthe news application, and may receive (e.g., from another computingdevice or any other desired entity) data indicating a news update forthe news application. The news update may include, for example, areal-time news feed, a periodic news update, a triggered newstransmission, or any other information associated with news.

In some embodiments, the virtual object is associated with a gamingapplication and the change associated with the virtual object involvesan occurrence of an event in the gaming application. A gamingapplication may refer to any application that may be configured toprovide a video game, a computer game, an electronic game, and/or anyother user interaction. A virtual object may include an icon, a widget,a symbol, a window, or any other user interface for the gamingapplication. At least one processor associated with the wearableextended reality appliance may implement the gaming application, and mayreceive (e.g., from another computing device, the gaming application, orany other desired entity) data indicating an occurrence of an event inthe gaming application. An event in the gaming application may include,for example, a message in a game (e.g., from another player), a missionin a game, a notification in a game, a status change in a game, or anyother information associated with a game.

In some embodiments, the change associated with the virtual object isunscheduled and the data reflecting the change associated with thevirtual object is received from a remote server. A remote server mayinclude any computing device that may be configured to transmitinformation. The remote server and the wearable extended realityappliance may be located in different rooms, in different buildings, indifferent cities, in different states, in different countries, or in twolocations having any desired distance therebetween. The changeassociated with the virtual object may be not appointed, assigned, ordesignated for a configured time. For example, the remote server maytransmit, to the wearable extended reality appliance, the datareflecting the change associated with the virtual object based on theoccurrence of an event that may be not subject to a configured schedule(e.g., an individual sending an email, an individual sending a textmessage, or a real-time news update). In some examples, the changeassociated with the virtual object may be unscheduled. For example, thedata reflecting the change associated with the virtual object may bereceived from another local software application, from a local sensor,from a remote software application, from a remote sensor, from a remoteprocessing device, and so forth. Additionally or alternatively, thechange associated with the virtual object may be scheduled (e.g.,expiration of a timer for a clock application associated with thevirtual object, an upcoming scheduled event for a calendar applicationassociated with the virtual object, or a scheduled transmission of anemail for an email application associated with the virtual object). Thedata reflecting the change associated with the virtual object may bereceived from an application associated with (e.g., underlying,supporting, or corresponding to) the virtual object or from any otherdesired entity.

With reference to FIG. 49 , at least one processor associated with thewearable extended reality appliance 4912 may, for example, receive datareflecting a change associated with the virtual object 4920. The datareflecting the change associated with the virtual object 4920 mayindicate, for example, that a new email is received by the emailapplication for which the virtual object 4920 may be the icon or widget.

Some embodiments involve determining whether the virtual object iswithin a field of view of the wearable extended reality appliance or isoutside the field of view of the wearable extended reality appliance. Afield of view may refer to a spatial extent that may be observed ordetected at any given moment. For example, a field of view of an entityas an observer or detector may include a solid angle via which theentity may be sensitive to radiation (e.g., visible light, infraredlight, or other optical signals). In some examples, a field of view mayinclude an angle of view. A field of view may be measured horizontally,vertically, diagonally, or in any other desired manner.

A field of view of the wearable extended reality appliance may refer to,for example, a portion, of the extended reality environment, associatedwith a display system of the wearable extended reality appliance at agiven moment. The display system of the wearable extended realityappliance may include, for example, an optical head-mounted display, amonocular head-mounted display, a binocular head-mounted display, asee-through head-mounted display, a helmet-mounted display, or any othertype of device configured to show images to a user. The portion of theextended reality environment may include a region or space where virtualcontent may be displayed by the display system of the wearable extendedreality appliance at a given moment. Additionally or alternatively, thefield of view of the wearable extended reality appliance may refer to aregional or spatial extent to which virtual content may be displayed bythe display system of the wearable extended reality appliance at a givenmoment. In some examples, from the perspective of a user wearing thewearable extended reality appliance, the field of view of the wearableextended reality appliance may be a solid angle via which the user mayview, perceive, observe, detect, and/or be sensitive to virtual contentas displayed (e.g., projected or radiated) by the display system of thewearable extended reality appliance (e.g., at a given moment).

At least one processor associated with the wearable extended realityappliance may determine whether the virtual object located in theextended reality environment is within the field of view of the wearableextended reality appliance or is outside the field of view of thewearable extended reality appliance, for example, using a ray castingalgorithm, using a rasterization algorithm, using a ray trackingalgorithm, and so forth. For example, a user wearing the wearableextended reality appliance may be not able to view the virtual object ifthe virtual object is outside the field of view of the wearable extendedreality appliance, as the virtual object may be not displayed by thedisplay system of the wearable extended reality appliance at the givenmoment. A user wearing the wearable extended reality appliance may beable to view the virtual object if the virtual object is within thefield of view of the wearable extended reality appliance, as the virtualobject may be displayed by the display system of the wearable extendedreality appliance at the given moment.

The determination of whether the virtual object is within or outside thefield of view of the wearable extended reality appliance may be made,for example, at or around the time of the change associated with thevirtual object, at or around the time of initiating a sensory prompt forthe change (e.g., the first or second sensory prompt as describedherein), at or around a time between the change and the initiating ofthe sensory prompt, at or around a selected time before the change,and/or at or around any other desired time for making the determination.In some examples, the determination of whether the virtual object iswithin or outside the field of view of the wearable extended realityappliance may be made in response to the receiving of the datareflecting the change associated with the virtual object. In someexamples, the determination of whether the virtual object is within oroutside the field of view of the wearable extended reality appliance maybe made in preparation for the initiating of a sensory prompt for thechange associated with the virtual object.

The at least one processor associated with the wearable extended realityappliance may determine that the virtual object is within the field ofview of the wearable extended reality appliance, for example, if acertain percentage larger than zero percent (e.g., 0.1 percent, 1percent, 20 percent, 50 percent, or 100 percent) of the virtual objectis within the field of view of the wearable extended reality appliance.The at least one processor may determine that the virtual object isoutside the field of view of the wearable extended reality appliance,for example, if the virtual object does not have the certain percentagewithin the field of view of the wearable extended reality appliance. Insome examples, the at least one processor associated with the wearableextended reality appliance may determine whether the virtual object iswithin or outside the field of view of the wearable extended realityappliance based on determining whether a particular point on the virtualobject is within or outside the field of view of the wearable extendedreality appliance. Additionally or alternatively, the at least oneprocessor associated with the wearable extended reality appliance maydetermine whether the virtual object is within or outside the field ofview of the wearable extended reality appliance in any other desiredmanner.

With reference to FIG. 49 , a field of view 4916 of the wearableextended reality appliance 4912 may be associated with a display systemof the wearable extended reality appliance 4912. The field of view 4916may move and/or rotate as the display system moves and/or rotates. At agiven moment, the display system may cause display, to the user 4910, ofvirtual content within the field of view 4916 and may not cause display,to the user 4910, of virtual content outside the field of view 4916. Thefield of view 4916 may be a region or space in the extended realityenvironment 4914 and/or may be a solid angle from a point of observationby the user 4910 (e.g., the eye(s) of the user 4910 and/or a point inthe area of or nearby the eye(s) of the user 4910).

At least one processor associated with the wearable extended realityappliance 4912 may determine whether the virtual object 4920 is withinthe field of view 4916 or is outside the field of view 4916, forexample, based on (e.g., in response to) the receiving of the datareflecting the change associated with the virtual object 4920. In someexamples, with reference to FIG. 49 , the at least one processorassociated with the wearable extended reality appliance 4912 maydetermine that the virtual object 4920 is within the field of view 4916.In some examples, with reference to FIG. 51 (which may illustrate one ormore elements as described in connection with FIG. 49 ), the at leastone processor associated with the wearable extended reality appliance4912 may determine that the virtual object 4920 is outside the field ofview 4916.

Some embodiments involve causing the wearable extended reality applianceto initiate a first sensory prompt indicative of the change associatedwith the virtual object when the virtual object is determined to bewithin the field of view. A sensory prompt may refer to any indicationthat may be configured to be sensed by an individual. A sensory promptmay relate to any sense of an individual, such as sight, smell, touch,taste, hearing, or any other ability of an individual to gatherinformation. In some examples, a sensory prompt may be used to provide anotification to a user. For example, a sensory prompt may include avisual notification, an audible notification, or a tactile notification.A computing device may cause a sensory prompt to be generated (e.g., viaone or more output devices, such as a screen, a speaker, or a vibrator),based on one or more triggering events, such as a change associated witha virtual object rendered by the computing device.

At least one processor associated with the wearable extended realityappliance may cause the wearable extended reality appliance to initiatea first sensory prompt indicative of the change associated with thevirtual object located in the extended reality environment when thevirtual object is determined to be within the field of view of thewearable extended reality appliance. Initiation of the first sensoryprompt may be based on receipt of data reflecting the change associatedwith the virtual object and/or may be based on a determination that thevirtual object is within the field of view of the wearable extendedreality appliance. The first sensory prompt may include, for example, anotification (e.g., visual, audible, or tactile) that may indicate thechange associated with the virtual object. The first sensory prompt mayinclude, for example, a popup notification on a physical or virtualscreen, a change of the appearance of the virtual object (e.g., addingor changing a mark, a red dot, a number, or any other indication on thevirtual object), a popup virtual object as a notification, or any otherdesired indication. The first sensory prompt may include, for example, apreview of at least a portion of the content of the change associatedwith the virtual object, a summary of the content of the changeassociated with the virtual object, an indication of the existence ofthe change associated with the virtual object, and/or an indication withany desired level of detail of the change associated with the virtualobject.

With reference to FIG. 50 (which may illustrate a use snapshot based onthe examples as described in connection with FIG. 49 ), at least oneprocessor associated with the wearable extended reality appliance 4912may cause the wearable extended reality appliance 4912 to initiate afirst sensory prompt indicative of the change associated with thevirtual object 4920 when the virtual object 4920 is determined to bewithin the field of view 4916. The initiating of the first sensoryprompt may be based on (e.g., in response to) the receiving of the datareflecting the change associated with the virtual object 4920 and/or thedetermination that the virtual object 4920 is within the field of view4916. The first sensory prompt may include a change of the appearance ofthe virtual object 4920 as displayed by the wearable extended realityappliance 4912. For example, appearing on the virtual object 4920, anumber indicating a quantity of emails received by the email applicationfor which the virtual object 4920 may be the icon or widget may changefrom “11” (as shown in FIG. 49 ) to “12” (as shown in FIG. 50 ). Thefirst sensory prompt (e.g., the change of the number from “11” to “12”)may indicate that a new email is received by the email application forwhich the virtual object 4920 may be the icon or widget.

Some embodiments involve causing the wearable extended reality applianceto initiate a second sensory prompt indicative of the change associatedwith the virtual object when the virtual object is determined to beoutside the field of view. In some embodiments, the second sensoryprompt differs from the first sensory prompt. At least one processorassociated with the wearable extended reality appliance may cause thewearable extended reality appliance to initiate a second sensory promptindicative of the change associated with the virtual object located inthe extended reality environment when the virtual object is determinedto be outside the field of view of the wearable extended realityappliance. The initiation of the second sensory prompt may be based onthe receipt of data reflecting a change associated with the virtualobject and/or may be based on the determination that the virtual objectis outside the field of view of the wearable extended reality appliance.The second sensory prompt may include, for example, a notification(e.g., visual, audible, or tactile) that may indicate the changeassociated with the virtual object. Additionally or alternatively, thesecond sensory prompt may include, for example, a popup notification ona physical or virtual screen, a change of the appearance of the virtualobject (e.g., adding or changing a mark, a red dot, a number, or anyother indication on the virtual object), a popup virtual object as anotification, or any other desired indication. The second sensory promptmay further include, for example, a preview of at least a portion of thecontent of the change associated with the virtual object, a summary ofthe content of the change associated with the virtual object, anindication of the existence of the change associated with the virtualobject, and/or an indication with any desired level of detail of thechange associated with the virtual object.

The second sensory prompt may be different from the first sensory prompt(e.g., for the same change associated with the virtual object or for thesame received data reflecting the change associated with the virtualobject). For example, for the change associated with the virtual objector the received data reflecting the change, the at least one processorassociated with the wearable extended reality appliance may initiate thefirst sensory prompt or the second sensory prompt based on whether thevirtual object is within or outside the field of view of the wearableextended reality appliance. The first sensory prompt and the secondsensory prompt may be different in terms of a type or category of asensory prompt, a quantity of sensory prompting items, a degree ofsensory prompting effect (e.g., disturbance), content of a sensoryprompt, or any other aspect of a sensory prompt. For example, the firstsensory prompt may be a visual notification, and the second sensoryprompt may be an audible notification. As another example, the firstsensory prompt may include one item of visual notification (e.g., achange of the appearance of the virtual object), and the second sensoryprompt may include two or more items of visual notification (e.g., apopup notification window and a change of the appearance and/or locationof the virtual object). As another example, the first sensory prompt maybe an audible notification with a lower degree of loudness, and thesecond sensory prompt may be an audible notification with a higherdegree of loudness. As another example, the first sensory prompt may bea visual notification with first content (e.g., a change of theappearance of the virtual object), and the second sensory prompt may bea visual notification with second content (e.g., a popup notificationwindow with a preview of the content of the change associated with thevirtual object). Additionally or alternatively, the first sensory promptand the second sensory prompt may be different in any other desiredmanner.

With reference to FIG. 52 (which may illustrate a use snapshot based onthe examples as described in connection with FIG. 51 ), at least oneprocessor associated with the wearable extended reality appliance 4912may cause the wearable extended reality appliance 4912 to initiate asecond sensory prompt indicative of the change associated with thevirtual object 4920 when the virtual object 4920 is determined to beoutside the field of view 4916 (as shown in FIG. 51 ). The initiating ofthe second sensory prompt may be based on (e.g., in response to) thereceiving of the data reflecting the change associated with the virtualobject 4920 and/or the determination that the virtual object 4920 isoutside the field of view 4916 (as shown in FIG. 51 ).

With reference to FIG. 52 , the second sensory prompt may include achange of the size of the virtual object 4920. For example, the size ofthe virtual object 4920 may be expanded (e.g., by 150 percent, by 200percent, by 300 percent, or to any other desired degree). In someexamples, expanding the size of the virtual object 4920 may make atleast a portion of the virtual object 4920 enter the field of view 4916,and thus cause at least the portion to be displayed by the wearableextended reality appliance 4912 to the user 4910. Additionally oralternatively, the second sensory prompt may include a change of theappearance of the virtual object 4920. For example, on the virtualobject 4920, a number indicating a quantity of emails received by theemail application for which the virtual object 4920 may be the icon orwidget may change from “11” (as shown in FIG. 51 ) to “12” (as shown inFIG. 52 ). Additionally or alternatively, the second sensory prompt mayinclude an audible notification. For example, at least one processorassociated with the wearable extended reality appliance 4912 may causean audible notification 5210 to be generated (e.g., using one or morespeakers associated with the wearable extended reality appliance 4912).The audible notification 5210 may sound as originating from the currentlocation of the virtual object 4920. The audible notification 5210 mayinclude, for example, a beep, a tone, an audio segment, an audiomessage, an audio associated with the change associated with the virtualobject 4920, or any other desired audio.

The second sensory prompt (e.g., including the expanding of the size ofthe virtual object 4920, the change of the number from “11” to “12,” andthe audible notification 5210) may indicate that a new email is receivedby the email application for which the virtual object 4920 may be theicon or widget. The second sensory prompt (e.g., including the expandingof the size of the virtual object 4920, the change of the number from “1I” to “12,” and the audible notification 5210) may differ from the firstsensory prompt (e.g., including the change of the number from “11” to“12”).

In some embodiments, the first sensory prompt includes at least one of avisual notification, an audible notification, or a tactile notification,and the second sensory prompt includes at least two of a visualnotification, an audible notification, or a tactile notification. Avisual notification may include any indication that may be viewed by auser. An audible notification may include any indication that may beheard by a user. A tactile notification may include any indication thatmay be perceived by a user with the sense of touch. In some examples,the second sensory prompt may include a larger quantity of notificationsthan the first sensory prompt. In some examples, the second sensoryprompt may include a same quantity of notifications as the first sensoryprompt. In some examples, the second sensory prompt may include asmaller quantity of notifications than the first sensory prompt.

In some embodiments, the second sensory prompt causes the virtual objectto move, and the first sensory prompt causes the virtual object tochange its appearance without moving. For example, the change of theappearance of the virtual object without moving (e.g., associated withthe first sensory prompt) may include, for example, adding or changing amark, a red dot, a number, or any other indication on the virtualobject, without changing the location of the virtual object in theextended reality environment. Causing the virtual object to move (e.g.,associated with the second sensory prompt) may include, for example,changing the location of the virtual object in the extended realityenvironment. In the second sensory prompt, the virtual object may becaused to move in any desired direction. In some embodiments, causingthe virtual object to move includes causing the virtual object totemporarily appear in the field of view of the wearable extended realityappliance. For example, the virtual object may move in a directiontowards the field of view of the wearable extended reality appliance,across the boundary of the field of view, and into the field of view. Insome examples, the virtual object may move into the field of view alonga surface on which the virtual object may be placed (e.g., a physical orvirtual wall, a physical or virtual whiteboard, or any other desiredsurface). Additionally or alternatively, the virtual object may move ina direction, towards the field of view, that may connect the location ofthe virtual object and a point, on the boundary of the field of view,that may be in proximity to (e.g., closest as measured in the space ofthe extended reality environment, or closest as measured on a particularsurface in the extended reality environment) the location of the virtualobject. In some examples, the virtual object may move to a location, inthe field of view, that may be in proximity to the boundary of the fieldof view (e.g., be in proximity to the point on the boundary). In someexamples, the virtual object may move to any desired location in thefield of view. The virtual object may stay at the location in the fieldof view for a temporary period of time (e.g., 0.5 seconds, 1 second, 2seconds, 3 seconds, 5 seconds, 10 seconds, or any other desired time).Additionally or alternatively, the virtual object may stay at thelocation in the field of view until a user interacts with the virtualobject.

In some embodiments, when the field of view of the wearable extendedreality appliance includes a virtual screen, the second sensory promptcauses a popup notification to be displayed on the virtual screen, andthe first sensory prompt causes the virtual object to change itsappearance in an absence of a popup notification on the virtual screen.A virtual screen may include a virtual object that may resemble aphysical screen. The virtual screen may be rendered by the wearableextended reality appliance and may be displayed in the field of view ofthe wearable extended reality appliance. The popup notification (e.g.,associated with the second sensory prompt) may include, for example, anyindication that may, in response to a triggering event, promptly appearin the foreground of the visual interface rendered by the wearableextended reality appliance. The popup notification may appear in anydesired location on the virtual screen. In some examples, the popupnotification may indicate information of the change associated with thevirtual object, such as a preview of the content of the changeassociated with the virtual object, a summary of the content of thechange associated with the virtual object, the existence of the changeassociated with the virtual object, or any other desired information.The change of the appearance of the virtual object (e.g., associatedwith the first sensory prompt) may include, for example, adding orchanging a mark, a red dot, a number, or any other indication on thevirtual object. In connection with the change of the appearance of thevirtual object (e.g., associated with the first sensory prompt), thewearable extended reality appliance may not cause display of a popupnotification on the virtual screen.

In some embodiments, the second sensory prompt is indicative of alocation of the virtual object outside the field of view of the wearableextended reality appliance. For example, the second sensory prompt maybe indicative of a direction between the location of the virtual objectand the field of view or a position or area within the field of view(e.g., by associating the second sensory prompt with an edge or locationof the field of view closest to the location of the virtual object, byassociating the second sensory prompt with a motion directed away fromthe location of the virtual object, by associating the second sensoryprompt with motion directed towards the location of the virtual object,and/or in any other desired manner). In some examples, the secondsensory prompt may be indicative of a distance between the location ofthe virtual object and the field of view or a position or area withinthe field of view. For example, one or more aspects (e.g., theintensity, the level of loudness, or the level of detail of the content)of the second sensory prompt may be configured based on (e.g.,proportional to or inversely proportional to) the distance.

In some embodiments, the second sensory prompt includes an audibleoutput configured to appear as originating from the location of thevirtual object outside the field of view of the wearable extendedreality appliance. An audible output may include, for example, anyindication that may be heard by a user. At least one processorassociated with the wearable extended reality appliance may cause theaudible output to be generated using a method that may create adirectional audible perspective. For example, stereophonic sound may beused to generate the audible output of the second sensory prompt, sothat a user of the wearable extended reality appliance may perceive theaudible output as originating from the location of the virtual objectoutside the field of view of the wearable extended reality appliance. Insome examples, the first sensory prompt may not include an audibleoutput configured to appear as originating from the location of thevirtual object. In some examples, both the first sensory prompt and thesecond sensory prompt may include audible outputs configured to appearas originating from the location of the virtual object, and the audibleoutput of the first sensory prompt may differ from the audible output ofthe second sensory prompt in at least one of a tone, a volume, a pitch,a duration, or any other aspect of an audible output. For example, theaudible output of the second sensory prompt may be more intense than theaudible output of the first sensory prompt.

In some embodiments, the second sensory prompt includes a visual outputconfigured to appear as originating from the location of the virtualobject outside the field of view of the wearable extended realityappliance. A visual output may include, for example, any indication thatmay be viewed by a user. The visual output may include, for example,ripples originating from the location of the virtual object and enteringthe field of view, a moving arrow or object originating from thelocation of the virtual object and entering the field of view, or anyother visual indication that may be displayed in the field of view asoriginating from the location of the virtual object. In some examples,the motion of the visual output may be along a surface on which thevirtual object may be placed. In some examples, the motion of the visualoutput may be along a path that may be associated with proximity (e.g.,closest) between the location of the virtual object and the field ofview. Additionally or alternatively, the motion of the visual output maybe determined in any other desired manner. In some examples, the firstsensory prompt may not include a visual output configured to appear asoriginating from the location of the virtual object. In some examples,both the first sensory prompt and the second sensory prompt may includevisual outputs configured to appear as originating from the location ofthe virtual object, and the visual output of the first sensory promptmay visually differ from the visual output of the second sensory prompt.For example, the visual output of the second sensory prompt may beremote from the location of the virtual object while the visual outputof the first sensory prompt may be local to the location of the virtualobject. In another example, the visual output of the second sensoryprompt may move in the extended reality environment while the visualoutput of the first sensory prompt may be local to a specific area ofthe extended reality environment (such as the area of the virtualobject). In yet another example, the visual output of the second sensoryprompt may include one graphical indication, while the visual output ofthe first sensory prompt may include a different graphical indication.In an additional example, the visual output of the second sensory promptmay include textual information, while the visual output of the firstsensory prompt may include different textual information.

Some embodiments involve estimating an importance of the changeassociated with the virtual object, and determining a degree ofdisturbance corresponding to the second sensory prompt based on theestimated importance of the change. For example, at least one processorassociated with the wearable extended reality appliance may estimate theimportance of the change associated with the virtual object based on oneor more of various factors, such as a message sender associated with thechange, an indicator of importance in a message associated with thechange, a date and time of the change, or any other relevant factor. Insome examples, to estimate the importance of the change, the at leastone processor may analyze the content of the change, for example, byusing natural language processing algorithms, voice recognitionalgorithms, or any other desired method. The at least one processor maydetermine a degree of disturbance corresponding to the second sensoryprompt based on the estimated importance of the change. The degree ofdisturbance corresponding to the second sensory prompt may include, forexample, a degree of loudness of an audible output, a displayed size ofa visual output, a vibration amplitude of a tactile output, an amount ofthe content of an output, or any other measurement of interruption ofthe second sensory prompt. In some examples, the degree of disturbancemay be proportional to the estimated importance of the change.

Some embodiments involve receiving image data; analyzing the image datato determine an activity of a user of the wearable extended realityappliance; estimating a relevancy level of the change associated withthe virtual object based on the determined activity of the user; anddetermining a degree of disturbance corresponding to the second sensoryprompt based on the relevancy level of the change. In some examples,image data may be captured using an image sensor included in or separatefrom the wearable extended reality appliance. The activity of the usermay be a physical activity of the user. The analysis of the image datamay include usage of a visual activity recognition algorithm or agesture recognition algorithm. Such algorithms may determinedegrees/amounts of bodily movement in image data by identifying a bodyor parts thereof in pixels and measuring pixel movement. In someexamples, the physical activity may include an interaction with aphysical object. An object recognition algorithm may be used to identifya type of the physical object, and the relevancy level may be determinedbased on the type of the physical object (e.g., based on an affinitybetween the type of the physical object and the virtual object or anaffinity between the type of the physical object and the changeassociated with the virtual object). In some examples, the physicalactivity of the user may include an interaction with a second virtualobject using hand gestures. The relevancy level may be determined basedon the second virtual object (e.g., based on an affinity between thesecond virtual object and the virtual object or an affinity between thesecond virtual object and the change associated with the virtualobject). The degree of disturbance corresponding to the second sensoryprompt may include, for example, a degree of loudness of an audibleoutput, a displayed size of a visual output, a vibration amplitude of atactile output, an amount of the content of an output, or any othermeasurement of interruption of the second sensory prompt. In someexamples, the degree of disturbance may be proportional to the relevancylevel of the change.

For example, if the physical object or the second virtual object is of awork type (e.g., a physical book, a physical file, a word processingprogram, a spreadsheet program, or a presentation program), and thevirtual object or the change associated with the virtual object is of agaming type (e.g., a video game), the relevancy level of the change maybe determined to be low. If the physical object or the second virtualobject is of a work type (e.g., a physical book, a physical file, a wordprocessing application, a spreadsheet application, or a presentationapplication), and the virtual object or the change associated with thevirtual object is of a work type (e.g., an email application), therelevancy level of the change may be determined to be high.

Some embodiments involve accessing a group of rules associating degreesof disturbance with degrees of virtual object changes, determining thatthe change associated with the virtual object corresponds to a specificrule of the group of rules, and implementing the specific rule to set adegree of disturbance corresponding to the second sensory prompt. Forexample, at least one processor associated with the wearable extendedreality appliance may store the group of rules in a memory associatedwith the wearable extended reality appliance. The at least one processormay access the group of rules, for example, in preparation forinitiating a sensory prompt (e.g., the second sensory prompt) for thechange associated with the virtual object. The degree of disturbancecorresponding to the second sensory prompt may include, for example, adegree of loudness of an audible output, a displayed size of a visualoutput, a vibration amplitude of a tactile output, an amount of thecontent of an output, or any other measurement of interruption of thesecond sensory prompt. The group of rules may map various virtual objectchanges (and/or the degrees of the changes) to corresponding degrees ofdisturbance of a sensory prompt. For example, each rule in the group ofrules may indicate a virtual object change (and/or a degree of thechange) and a corresponding degree of disturbance of a sensory promptfor the change. The group of rules may be configured, for example, by auser, an administrator, or any other desired entity. Based on accessingthe group of rules, the at least one processor may, for example, searchin the group of rules using an identifier of the change associated withthe virtual object as a search key. The at least one processor mayidentify the specific rule based on the searching, and may implement thespecific rule to set the degree of disturbance corresponding to thesecond sensory prompt. The degree of disturbance corresponding to thesecond sensory prompt may be set to, for example, the degree ofdisturbance as indicated in the specific rule.

Some embodiments involve halting the second sensory prompt upondetection of a trigger. For example, at least one processor associatedwith the extended reality appliance may halt (e.g., stop, suspend, orend) the second sensory prompt based on detecting a trigger. The triggermay include, for example, user interaction with and/or user attention tothe virtual object, the change associated with the virtual object,and/or the second sensory prompt, a command to halt the second sensoryprompt, a command to mute notifications, an expiration of a timer fordisplaying the second sensory prompt, or any other desired event.

In some embodiments, detection of the trigger includes identifying entryof the virtual object into the field of view of the wearable extendedreality appliance. For example, based on the second sensory prompt, auser of the wearable extended reality appliance may direct the user'sattention to the virtual object, for example, by moving and/or rotatingthe wearable extended reality appliance so that the field of view of thewearable extended reality appliance may cover the location of thevirtual object and the virtual object may enter into the field of view.At least one processor associated with the wearable extended realityappliance may detect the entry of the virtual object into the field ofview and may, based on the detected entry, halt the second sensoryprompt.

Some embodiments involve analyzing input received from a sensorassociated with the wearable extended reality appliance to detect thetrigger for halting the second sensory prompt. The sensor associatedwith the wearable extended reality appliance may include, for example,an image sensor, an eye-tracking sensor, a sensor for trackinghead-motion, a sensor included in or associated with an input device, orany other desired device for receiving information from a user. At leastone processor associated with the wearable extended reality appliancemay analyze input received from the sensor to detect the trigger forhalting the second sensory prompt. The analysis of the input to detectthe trigger may include, for example, detecting user interaction withand/or user attention to the virtual object, the change associated withthe virtual object, and/or the second sensory prompt, detecting acommand to halt the second sensory prompt, detecting a command to mutenotifications, or detecting any other desired event as the trigger.

Some embodiments involve, after the wearable extended reality applianceinitiated the second sensory prompt, upon entry of the virtual objectinto the field of view of the wearable extended reality appliance,causing the wearable extended reality appliance to initiate the firstsensory prompt indicative of the change associated with the virtualobject. For example, based on the second sensory prompt, a user of thewearable extended reality appliance may direct the user's attention tothe virtual object, for example, by moving and/or rotating the wearableextended reality appliance so that the field of view of the wearableextended reality appliance may cover the location of the virtual objectand the virtual object may enter into the field of view. At least oneprocessor associated with the wearable extended reality appliance maydetect the entry of the virtual object into the field of view and may,based on the detected entry, cause the wearable extended realityappliance to initiate the first sensory prompt indicative of the changeassociated with the virtual object (and may halt the second sensoryprompt). For example, the second sensory prompt may include a displayedvirtual arrow towards the location of the virtual object, or a displayedillustration (e.g., a preview visual notification) of the changeassociated with the virtual object (e.g., a received email), and thefirst sensory prompt may include a change of the appearance of thevirtual object (e.g., a change of a number, shown on the virtual object,indicating the quantity of received emails).

Some embodiments involve receiving real-time movement data associatedwith the wearable extended reality appliance; analyzing the real-timemovement data to determine a prospective entrance of the virtual objectinto the field of view of the wearable extended reality appliance; andin response to the determined prospective entrance of the virtual objectinto the field of view of the wearable extended reality appliance,withholding causing the wearable extended reality appliance to initiatethe second sensory prompt. The real-time movement data may indicatemovement of the wearable extended reality appliance and/or a componentor element of the wearable extended reality appliance. The real-timemovement data may include, for example, data captured using an inertiasensor (such as an accelerometer and/or a gyroscope) included in orseparate from the wearable extended reality appliance. Additionally oralternatively, the real-time movement data may be obtained by analyzingimages captured using an image sensor included in or separate from thewearable extended reality appliance (e.g., using an ego-motionalgorithm). In some examples, the real-time movement data may becaptured repeatedly, continuously, or periodically using one or more ofvarious desired sensors and may be processed in real-time.

At least one processor associated with the wearable extended realityappliance may analyze the real-time movement data to determine theprospective entrance of the virtual object into the field of view of thewearable extended reality appliance. The at least one processor maydetermine that a current movement may cause a prospective entrance ofthe virtual object into the field of view, for example, if the virtualobject may be predicted to enter the field of view within a selectedtime period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.3 seconds,or any other desired time) from the current time. The at least oneprocessor may make the determination, for example, based on the currentposition and/or orientation of the wearable extended reality applianceand a current speed and direction of moving and/or rotating of thewearable extended reality appliance. The at least one processor may makethe determination (e.g., repeatedly, continuously, or periodically), forexample, when the virtual object may be currently outside the field ofview. In some examples, the at least one processor may use historicaldata to predict whether there may be a prospective entrance of thevirtual object into the field of view. For example, the at least oneprocessor may store parameter information (e.g., the position of thewearable extended reality appliance, the orientation of the wearableextended reality appliance, the speed of moving and/or rotating of thewearable extended reality appliance, the direction of moving and/orrotating of the wearable extended reality appliance, and/or any otherrelevant information) during a time period before a detected actualentrance of the virtual object into the field of view, and may comparethe stored parameter information with currently gathered parameterinformation (e.g., for the time period before the current time). Basedon the comparison, the at least one processor may determine (e.g.,predict) a prospective entrance of the virtual object into the field ofview, for example, if a degree of similarity between the storedparameter information and the currently gathered parameter information(e.g., a confidence score) satisfies (e.g., meets or exceeds) aconfigured or selected threshold level of similarity. Additionally oralternatively, the at least one processor may determine (e.g., predict)a prospective entrance of the virtual object into the field of view inany other desired manner.

In response to the determined prospective entrance of the virtual objectinto the field of view of the wearable extended reality appliance, theat least one processor associated with the wearable extended realityappliance may withhold causing the wearable extended reality applianceto initiate the second sensory prompt. The withholding may last for anydesired time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.3seconds, or any other desired time). In some examples, the withholdingmay last for a time period that may be same as or may approximate thetime period within which the virtual object may be predicted (e.g., bythe at least one processor) to enter the field of view. If the virtualobject does not enter the field of view during the withholding timeperiod, the at least one processor may, based on the expiration of thewithholding time period, cause the wearable extended reality applianceto initiate the second sensory prompt. If the virtual object enters thefield of view during the withholding time period, the at least oneprocessor may, based on the entrance of the virtual object into thefield of view, cause the wearable extended reality appliance to initiatethe first sensory prompt.

Some embodiments involve a system for initiating location-driven sensoryprompts reflecting changes to virtual objects, the system comprising atleast one processor programmed to: enable interaction with a virtualobject located in an extended reality environment associated with awearable extended reality appliance; receive data reflecting a changeassociated with the virtual object; determine whether the virtual objectis within a field of view of the wearable extended reality appliance oris outside the field of view of the wearable extended reality appliance;cause the wearable extended reality appliance to initiate a firstsensory prompt indicative of the change associated with the virtualobject when the virtual object is determined to be within the field ofview; and cause the wearable extended reality appliance to initiate asecond sensory prompt indicative of the change associated with thevirtual object when the virtual object is determined to be outside thefield of view, wherein the second sensory prompt differs from the firstsensory prompt.

Some embodiments involve a method for initiating location-driven sensoryprompts reflecting changes to virtual objects, the method comprising:enabling interaction with a virtual object located in an extendedreality environment associated with a wearable extended realityappliance; receiving data reflecting a change associated with thevirtual object; determining whether the virtual object is within a fieldof view of the wearable extended reality appliance or is outside thefield of view of the wearable extended reality appliance; causing thewearable extended reality appliance to initiate a first sensory promptindicative of the change associated with the virtual object when thevirtual object is determined to be within the field of view; and causingthe wearable extended reality appliance to initiate a second sensoryprompt indicative of the change associated with the virtual object whenthe virtual object is determined to be outside the field of view,wherein the second sensory prompt differs from the first sensory prompt.

FIG. 53 is a flowchart illustrating an exemplary process 5300 forinitiating sensory prompts for changes based on a field of viewconsistent with some embodiments of the present disclosure. Withreference to FIG. 53 , in step 5310, instructions contained in anon-transitory computer-readable medium when executed by a processor maycause the processor to enable interaction with a virtual object locatedin an extended reality environment associated with a wearable extendedreality appliance. In step 5312, instructions contained in anon-transitory computer-readable medium when executed by a processor maycause the processor to receive data reflecting a change associated withthe virtual object. In step 5314, instructions contained in anon-transitory computer-readable medium when executed by a processor maycause the processor to determine whether the virtual object is within afield of view of the wearable extended reality appliance or is outsidethe field of view of the wearable extended reality appliance. In step5316, instructions contained in a non-transitory computer-readablemedium when executed by a processor may cause the processor to cause thewearable extended reality appliance to initiate a first sensory promptindicative of the change associated with the virtual object when thevirtual object is determined to be within the field of view. In step5318, instructions contained in a non-transitory computer-readablemedium when executed by a processor may cause the processor to cause thewearable extended reality appliance to initiate a second sensory promptindicative of the change associated with the virtual object when thevirtual object is determined to be outside the field of view, whereinthe second sensory prompt differs from the first sensory prompt.

Various embodiments may be described with reference to a system, method,apparatuses, and/or computer readable medium for performing orimplementing operations for selectively controlling a display of digitalobjects. It is intended that the disclosure of one is a disclosure ofall. For example, it is to be understood that the disclosure of one ormore processes embodied in a non-transitory computer-readable medium, asdescribed herein, may also constitute a disclosure of methodsimplemented by the computer readable medium, as well as systems and/orapparatuses for implementing processes embodied in the non-transitorycomputer-readable medium, for example, via at least one processor. Thus,in some embodiments, a non-transitory computer readable medium containsinstructions that when executed by at least one processor cause the atleast one processor to perform operations for selectively controlling adisplay of digital objects. Some aspects of such processes may occurelectronically over a network that may be wired, wireless, or both.Other aspects of such processes may occur using non-electronic means. Inthe broadest sense, the processes disclosed herein are not limited toparticular physical and/or electronic instrumentalities; rather, theymay be accomplished using any number of differing instrumentalities.

The term “non-transitory computer-readable medium” may be understood asdescribed earlier. The term “instructions” may refer to program codeinstructions that may be executed by a computer processor, for example,software instructions, computer programs, computer code, executableinstructions, source code, machine instructions, machine languageprograms, or any other type of directions for a computing device. Theinstructions may be written in any type of computer programminglanguage, such as an interpretive language (e.g., scripting languagessuch as HTML and JavaScript), a procedural or functional language (e.g.,C or Pascal that may be compiled for converting to executable code),object-oriented programming language (e.g., Java or Python), logicalprogramming language (e.g., Prolog or Answer Set Programming), or anyother programming language. In some embodiments, the instructions mayimplement methods associated with machine learning, deep learning,artificial intelligence, digital image processing, and any othercomputer processing technique.

In some embodiments, the instructions contained in the non-transitorycomputer-readable medium may include (e.g., embody) various processesfor selectively controlling a display of digital objects via a physicaldisplay of a computing device such as, for example, a wearable extendedreality appliance, including generating a plurality of digital objectsfor display, determining a usage status of a wearable extended realityappliance, selecting a display mode of a computing device, determiningto display and/or not display digital objects, outputting digitalobjects for presentation, presenting digital objects, causing at leastone digital object to appear and/or disappear from display, identifyinga change in a usage status of a wearable extended reality appliance,updating a display mode selection, revising a presentation of digitalobjects, and/or any process related to controlling a display of digitalobjects based on a usage of a wearable extended reality appliance, asdescribed herein. As used herein, a “computing device” includes anyelectronic component or group of components for manipulating data.Examples of computing devices include wearable extended reality orvirtual reality appliances, personal computers, laptops, servers,tablets, smart phones, smart watches, or any other device that includesat least one processor.

At least one processor may be configured to execute instructionscontained in the non-transitory computer-readable medium to causevarious processes to be performed for implementing operations forselectively controlling a display of digital objects, as describedherein. The term processor may be understood as described earlier. Forexample, the at least one processor may be one or more of server 210 ofFIG. 2 , mobile communications device 206, processing device 360 of FIG.3 , processing device 460 of FIG. 4 , processing device 560 of FIG. 5 ,and the instructions may be stored at any of memory devices 212, 311,411, or 511, or a memory of mobile device 206.

Disclosed embodiments may relate to operations for selectivelycontrolling display of digital objects. As used herein, the term“digital objects” may include, or otherwise denote, any type of datarepresentation or visual presentation generated by and/or presented byat least one computer or processing device. Digital objects may include,for example, any data or media (e.g., alphanumerical text, image data,audio data, video data) formatted for presenting information to a uservia, for example, an interface of an electronic device. “Display ofdigital objects” include the presentation of digital objects to a uservia one or more presentation devices. “Operations for selectivelycontrolling display of digital objects” may include one or more acts ofregulating, integrating, implementing, presenting, manipulating, and/orchanging a presentation of, at least one digital object, or group ofdigital objects. These digital objects may be capable of being presentedto a user via at least one appropriate physical display and/or extendedreality appliance at or for a particular time. Moreover, the digitalobjects may be presented to a user in response to at least one action,option, and/or environment that may be distinguishable from some otheraction, option, and/or environment. For example, operations forselectively controlling, or otherwise implementing selective controlover, a display of digital objects may relate to determining,regulating, managing, changing, and/or otherwise affecting the location,arrangement, appearance, status, accessibility, and/or overallpresentation of any number of digital objects via a physical displayand/or extended reality appliance, such as a wearable extended realityappliance. In some embodiments, operations for selectively controlling adisplay of digital objects may be performed within and/or between a realenvironment, a virtual environment, or real and virtual combinedenvironments for displaying digital objects to a user.

In some embodiments, the manner in which digital objects are displayedto a user may relate to the location, arrangement, appearance, status,accessibility, and/or overall presentation of a digital object or groupof digital objects as presented to a user via an extended realityappliance or physical display and/or combination of an extended realityappliance and/or physical display. In some embodiments, the manner inwhich digital objects are displayed may be selectively controlled tochange at or for a particular time in response to at least one action,option, and/or environment.

The display of digital objects may occur in a real environment via atleast one physical display for presenting digital content. In otherexamples, the display of digital objects may occur in an extendedreality environment, such as an augmented reality environment or a mixedreality environment, via a physical display of a computing device and/ora wearable extended reality appliance. For example, a wearable extendedreality appliance may be configured to enable a user of the wearableextended reality appliance to view the overall display of digitalobjects across multiple displays including a physical display forpresenting digital content and an extended reality appliance forpresenting virtual digital content.

Digital objects may be displayed to a user via at least one physicaldisplay of a computing device and/or a wearable extended realityappliance. In some embodiments, the wearable extended reality appliancemay be in communication with the at least one computing device. In someembodiments, digital objects may include, for example, at least oneapplication, widget, document, cursor, menu, option in a menu, at leastone icon which may activate a script for causing an action associatedwith the particular digital object associated with the icon and/orotherwise linked to related programs or applications, and/or any otherdata representation or visual presentation displayed, or configured fordisplay, via a physical display and/or via an extended realityappliance. In some embodiments, the digital objects may include realdigital objects and/or virtual digital objects. As described herein, areal digital object may relate to any digital object displayed to a uservia at least one physical display of a computing device. A virtualdigital object may relate to any digital object displayed to a user viaa wearable extended reality appliance. For example, at a particulartime, one digital object may be presented as a virtual digital objectvia a wearable extended reality appliance, as a real digital object viaa physical display, or as a virtual digital object and a real digitalobject simultaneously via an extended reality appliance and a physicaldisplay.

In some embodiments, a real digital object may include any graphictwo-dimensional digital content, graphic three-dimensional digitalcontent, inanimate digital content, animate digital content configuredto change over time or in response to triggers, and/or any other digitalcontent configured to be displayed to a user via a physical display. Forexample, real digital objects displayed via a physical display mayinclude a document, a widget inside a menu bar, and images. In someembodiments, a virtual digital object may include any inanimate virtualcontent, animate virtual content configured to change over time or inresponse to triggers, virtual two-dimensional content, virtualthree-dimensional content, a virtual constructive or destructive overlayover a portion of a physical and/or digital environment or over aphysical and/or real digital object, a virtual addition to a physicaland/or digital environment or to a physical and/or real digital object,and/or any other digital content configured to be displayed to a uservia a wearable extended reality appliance. For example, virtual digitalobjects displayed via an extended reality appliance may include avirtual document, virtual widgets inside a virtual menu bar, a virtualworkspace, and a realistic three-dimensional rendition of an image.

In some embodiments, a user may be able to interact with the digitalobjects, including real digital objects and/or virtual digital objects,presented via the physical display and/or the wearable extended realityappliance. In some embodiments, at least one real digital object may berelated, linked, associated, or otherwise correspond to at least onevirtual digital object, or vice versa. For example, the at least onereal digital object and the at least one virtual digital object mayshare at least one common feature and/or function. In one example,interaction with at least one virtual digital object may affect at leastone related, linked, or associated real digital object, and vice versa.In another example, interaction with at least one virtual digital objectmay not affect a related, linked, or associated real digital object, andvice versa. In some examples, at least one real digital object may beconverted to at least one virtual digital object, or vice versa, withinan extended reality environment at or for a particular time in responseto at least one action, option, and/or environment.

Computations provided by a computing device may include arithmeticand/or logic operations with or without human intervention. For example,a computing device may include one or more input devices, processingdevices for processing data instructions, output devices, and/or storagedevices for data and storage retention. In some embodiments, a computingdevice may relate to a standalone unit and/or a combination of relatedor interconnected units. In some embodiments, the computing device maybe directly or indirectly connected to a physical display, and/or may bea part of the physical display. Additionally, or alternatively, thecomputing device may be directly or indirectly connected to a wearableextended reality appliance and/or may be a part of the wearable extendedreality appliance. In some embodiments, the computing device may enablea user to interact with one or more digital objects within an extendedreality environment via a wearable extended reality appliance and/or viaanother device in communication with the computing device and/or withthe wearable extended reality appliance. The computing device may becapable of selectively controlling a display of one or more digitalobjects, consistent with some embodiments of the present disclosure. Inone example, the computing device may be configured to generate some orall of the digital objects for display via the physical display and/orthe wearable extended reality appliance. For example, a computing devicemay include a laptop computer, a desktop computer, a smartphone, awearable computer such as a smartwatch, and a tablet computer.

As used herein, the term “generating a plurality of digital objects fordisplay with use of a computing device” includes constructing and/orrendering of any number of digital objects for presentation a computingdevice. In some embodiments, the computing device may be configured togenerate one or more digital objects for display based on receivedand/or processed digital signals and/or any other form of data receivedand/or stored by the computing device. For example, digital signalsand/or data processed by the computing device may be used to presentdigital content, including real digital objects, to a user via aphysical display. Additionally, or alternatively, digital signals and/ordata processed by the computing device may be used to present virtualdigital content, including virtual digital objects, to a user via awearable extended reality appliance. In one example, digital signalsand/or data may indicate an appropriate position and/or angle of aviewpoint of a digital object such that digital content may be generatedfor display to the user at a particular position and/or angle within aparticular environment (e.g., a real environment or extended realityenvironment). In another example, digital signals and/or data mayindicate an appropriate presentation and/or appearance of a digitalobject such that the digital object has a particular presentation and/orappearance.

By way of example, FIG. 54 illustrates one non-limiting example of aplurality of digital objects presented to a user within an extendedreality environment, consistent with some embodiments of the presentdisclosure. FIG. 54 is a representation of just one embodiment, and itis to be understood that some illustrated elements and/or features mightbe omitted, and others added within the scope of this disclosure. Asshown, a user 5410 is wearing a wearable extended reality appliance 5411and sitting behind table 5413 supporting a keyboard 5414, mouse 5415,and computing device 5416 having physical display 5417.

The physical display 5417 of the computing device 5416 may be a desktopcomputer configured to display digital content to user 5410, forexample, real digital objects 5418A, 5418B, and 5418C. Real digitalobjects 5418A and 5418B are programs open on the physical display 5417and real digital object 5418C may be a cursor for interacting withdigital objects displayed via the physical display 5417 and controllableusing mouse 5415. While physical display 5417 of computing device 5416is depicted as a desktop computer, it is to be understood that, in someembodiments, the physical display may relate to any display orcombination of physical displays configured to display real digitalobjects to user 5410. Some non-limiting examples of such physicaldisplays may include a physical display of a laptop computer, a physicaldisplay of a tablet, a physical display of a smartphone, a physicaldisplay of a television, and so forth. In some examples, a physicaldisplay may be or include a device converting digital signals and/oranalog signals to perceptible light patterns. Additionally, whilekeyboard 5414 and mouse 5415 are depicted here as a wireless keyboardand a wireless mouse connected to computing device 5416 of physicaldisplay 5417, it is to be understood that the computing device may beindirectly or directly connected to, or in communication with, anynumber of peripheral devices.

As shown, wearable extended reality appliance 5411 may be a pair ofsmart glasses. The wearable extended reality appliance 5411 may beconnected via a wire to keyboard 5414 which may be in communication withthe computing device 5416. Wearable extended reality appliance 5411 maybe configured to display virtual digital content to user 5410 within anextended reality environment viewable through wearable extended realityappliance 5411. For example, virtual digital object 5419A and virtualdigital object 5419B are displayed to user 5410 via wearable extendedreality appliance 5411. From the perspective of user 5410, virtualdigital object 5419A is displayed next to physical display 5417 ofcomputing device 5416 and virtual digital object 5419B is displayed ontable 5413.

Some embodiments involve generating a plurality of digital objects fordisplay in connection with use of a computing device. In one example,generating a digital object for display may include selecting and/orgenerating one or more visuals associated with the digital object. Inanother example, generating a digital object for display may includeselecting and/or generating textual data for display in association withthe digital object. In another example, generating a digital object fordisplay may include selecting at least one of color, texture, size,position, orientation, illumination, intensity or opacity for thedisplay of at least part of the virtual object. In some examples,generating a digital object for display may include rendering thevirtual object, for example using a ray casting algorithm, using a raytracking algorithm, using rasterization algorithm, and so forth. Forexample, the digital object may be rendered for display using a singledisplay, using a stereo display, and so forth. For example, the digitalobject may be rendered from geometrical information associated with thevirtual object, from a three-dimensional model associated with thevirtual object, from a two-dimensional model associated with the virtualobject, from visuals associated with the virtual object, from textualinformation associated with the virtual object, and so forth.

In some embodiments, the computing device is operable in a first displaymode and in a second display mode. The term “display mode” may include aconfiguration or manner of operation. For example, particular displayparameters may apply to a particular mode. The particular mode may beapplied to a real and/or extended reality environment at a particulartime or for a particular duration. For example, a display parameter mayinclude any characteristic capable of defining or classifying the mannerin which any number of digital objects are displayed (e.g., location,arrangement, appearance, status, accessibility, and/or overallpresentation) to a user via a physical display and/or extended realityappliance. In some embodiments, the display mode in which the computingdevice operates may be based on a particular type of one or more of thedigital objects (e.g., real digital objects and/or virtual digitalobjects). In one example, the computing device may be configured tooperate in a display mode capable of presenting real digital objects fordisplay via at least one physical display. In another example, thecomputing device may be configured to operate in a display mode capableof presenting virtual digital objects for display via a wearableextended reality appliance. In yet another example, the computing devicemay be configured to operate in another display mode capable ofsimultaneously presenting digital content for display via at least onephysical display and virtual digital content for display via a wearableextended reality appliance.

As used herein, the term “operable” refers to an ability to work orperform. For example, a computing device may perform in multiple waysand may be switchable between modes of operation. Switching, forexample, may alter the way, manner, type or content of presented. In onedisplay mode, content may be displayed to the user via a physicaldisplay and in another mode, the content may be presented via a wearableextended reality appliance. Or, by way of another example, the computingdevice may be configured to switch between a first display mode forpresenting digital objects via the physical display and a second displaymode for presenting digital objects via the physical display and/or thewearable extended reality appliance. Additionally, or alternatively, insome embodiments, the computing device may be configured to operate in,and switch between, display modes that are different from the firstdisplay mode and the second display mode. In some embodiments, thecomputing device may be configured to switch between display modes inreal time or near real time in response to at least one action, option,and/or environment.

According to some embodiments, when the computing device is in the firstdisplay mode, the plurality of digital objects are displayed via aphysical display connected to the computing device. In a general sense,the term “displayed via a physical display” may relate to thepresentation of digital content, including real digital objects, to auser via a physical display which the user may perceive and/or interactwith in a real environment and/or an extended reality environment. Insome examples, a “physical display” may relate to any type of device orsystem that is directly or indirectly connected to a computing deviceand configured to present inanimate and/or animate graphictwo-dimensional and/or three-dimensional digital content to a user. Insome examples, a “physical display” may relate to any type of device orsystem that is configured to present, based on received digital and/oranalog signals, graphical information perceptible without using extendedreality equipment, such as a wearable extended reality appliance. Insome examples, digital objects presented via the physical display (e.g.,real digital objects) are perceived as objects positioned at thephysical location of the physical display, while digital objectspresented via the wearable extended reality appliance (e.g., virtualdigital objects) are perceived (e.g., due to optical illusion) asobjects positioned away of the wearable extended reality appliance. Insome examples, a physical display may relate to a non-transparentdisplay, while the wearable extended reality appliance may include oneor more transparent displays and may use the one or more transparentdisplays to present virtual digital objects in an optical illusioncausing the virtual digital objects to appear at a select position inthe environment away from the wearable extended reality appliance. Insome examples, a physical display may relate to a non-transparentdisplay, while the wearable extended reality appliance may include oneor more projectors and may use the one or more projectors to presentvirtual digital objects in an optical illusion causing the virtualdigital objects to appear at a select position in the environment awayfrom the wearable extended reality appliance.

In one embodiment, a physical display may include a computer screen,laptop screen, tablet screen, smartphone screen, projector screen,and/or any physical device capable of presenting digital content, suchas a plurality of real digital objects, to a user. The physical displaymay include one physical display or a combination of physical displaysin which at least one physical display is in direct or indirectcommunication with at least another physical display. Additionally, oralternatively, the physical display may include a combination ofdiscrete physical displays that are not in communication with oneanother.

As used herein, the “first display mode” may relate to the mode ofoperation of the computing device in which the computing device may beconfigured to generate and/or output digital content, such as aplurality of real digital objects, for display to a user via at leastone physical display in communication with the computing device. In someexamples, the visual presentation of at least one real digital objectgenerated by the computing device in the first display mode may beproduced in at least one confined region of the physical display. Forexample, when the computing device is operating in the first displaymode, real digital objects may be generated and/or presented to a uservia the physical display within any number of discrete and/or connectedsubsets of space within the entire space of the physical display. Insome embodiments, when the computing device is operating in the firstdisplay mode, the plurality of digital objects are not displayed to theuser via the wearable extended reality environment.

By way of a non-limiting example, FIG. 55A illustrates a user 5512 of acomputing device 5514 operating in a first display mode 5510. User 5512is shown sitting in front of a computing device 5514 operating in afirst display mode 5510. The computing device 5514 includes physicaldisplay 5515, keyboard 5516, and mouse 5517. The physical display 5515of computing device 5514 is configured to display digital content, suchas a plurality of real digital objects 5519A to 5519C, to user 5512. Inthis non-limiting embodiment, computing device 5514 is depicted as alaptop, but as described above, computing device 5514 may be any type ofdevice or combination of devices capable of operating in a first displaymode and a second display mode. Moreover, while physical display 5515 ofcomputing device 5514 is depicted here as a laptop screen, it is to beunderstood that a computing device 5514 may be indirectly or directlyconnected to, or in communication with, any physical display orcombination of physical displays configured to display digital content,such as a plurality of real digital objects, to the user 5512.Additionally, while keyboard 5516 and mouse 5517 are depicted here as akeypad and trackpad built into computing device 5514, it is to beunderstood that the computing device may be indirectly or directlyconnected to, or in communication with, any number of peripheral devicesincluding a wireless keyboard, a wireless mouse, and a wearable extendedreality appliance.

The digital content displayed to user 5512 by physical display 5515 ofcomputing device 5514 operating in the first display mode 5510 includes,for example, a cursor 5518A and a plurality of real digital objects5519A to 5519C. Additionally, it is to be understood that cursor 5518Amay constitute a digital object, such as a real digital object, asdescribed herein. In the first display mode, cursor 5518A may moveanywhere within physical display 5515 and may interact with any digitalcontent displayed therein, such as real digital object 5519A, realdigital object 5519B, real digital object 5519C, and/or any group orsub-group of real digital objects. For example, user 5512 may interactwith applications or widgets, such as real digital objects 5519A, realdigital object 5519B, and real digital object 5519C, displayed inphysical display 5515 using keyboard 5516 and/or mouse 5517.

According to some embodiments, when the computing device is in thesecond display mode, some of the plurality of digital objects aredisplayed via the physical display, and at least one other of theplurality of digital objects is displayed via a wearable extendedreality appliance. In a general sense, the term “displayed via awearable extended reality appliance” may relate to the presentation ofvirtual digital content, including virtual digital objects describedabove, to a user which the user may perceive, and/or interact with, inan extended reality environment via the wearable extended realityappliance. In one example, textual content entered using a keyboard (forexample, using a physical keyboard, using a virtual keyboard, etc.) maybe presented via the wearable extended reality appliance in real time asthe textual content is typed. In another example, a virtual cursor maybe presented via the wearable extended reality appliance, and thevirtual cursor may be controlled by a pointing device (such as aphysical pointing device, a virtual pointing device, a computer mouse, ajoystick, a touchpad, a physical touch controller, and so forth). In yetanother example, virtual displays, including one or more windows of agraphical user interface operating system, may be presented via thewearable extended reality appliance.

In some embodiments, virtual digital objects generated by the computingdevice may be displayed via a wearable extended reality appliance in atleast one virtual region of the extended reality environment. Forexample, virtual digital objects may be generated and/or presented to auser via the wearable extended reality appliance within any number ofdiscrete and/or connected subsets of space within the entire space ofthe extended reality environment. A subset of space may relate to atwo-dimensional or three-dimensional space within the extended realityenvironment that may be fixed relative to a particular physical objectand/or digital object, fixed relative to a part of the extended realityappliance, or not fixed relative to any particular physical object,digital object, or part of the extended reality appliance. In someembodiments, virtual digital objects generated by the computing devicemay be displayed via the wearable extended reality appliance such thatthe dimensional orientation of at least one virtual digital objectwithin the extended reality environment is different from anothervirtual digital object, as viewed from the perspective of the user. Forexample, the perceived dimensional orientation of any one subset ofspace including at least one virtual digital object may be differentfrom, or similar to, the perceived dimensional orientation of any othersubset of space including at least another virtual digital object withinthe entire space of an extended reality environment. In one example, atleast two virtual digital objects may appear to exist within the sameplane of the extended reality environment. In another example, at leastone virtual digital object may appear to exist in a first plane of theextended reality environment, and at least another virtual digitalobject may appear to exist in a second plane of the extended realityenvironment that may intersect, or be parallel to, the first plane ofthe extended reality environment.

As used herein, the term “wearable extended reality appliance” may beunderstood as described earlier. In some embodiments, the wearableextended reality appliance may be directly or indirectly incommunication with the computing device and may include one wearableextended reality appliance or a combination of wearable extended realityappliances.

As used herein, the “second display mode” may relate to the mode ofoperation of the computing device in which the computing device maygenerate and/or output digital content, such as a plurality of digitalobjects, for display to the user via at least one physical display incommunication with the computing device and/or at least one wearableextended reality appliance. In some embodiments, in the second displaymode, the wearable extended reality appliance may present digitalcontent that may also be capable of being presented via the physicaldisplay. Additionally, or alternatively, the wearable extended realityappliance may present digital content that may not be capable of beingpresented via the physical display. In some embodiments, in the seconddisplay mode, the wearable extended reality appliance may display atleast one virtual digital object mimicking and/or extending thefunctionality of at least one real digital object displayed, orpreviously displayed, via the physical display. Additionally, oralternatively, the wearable extended reality appliance may display atleast one virtual digital object that is not related to thefunctionality of at least one real digital object displayed, orpreviously displayed, via the physical display.

By way of a non-limiting example, FIG. 55B illustrates a user 5512 of awearable extended reality appliance 5513 and a computing device 5514operating in a second display mode 5511. The computing device 5514illustrated herein is consistent with the computing device 5514operating in the first display mode 5510 illustrated in FIG. 55A. Asshown, the wearable extended reality appliance 5513 is in wirelesscommunication with the computing device 5514 and is configured todisplay virtual digital content to the user 5512 when the computingdevice is in the second display mode 5511. Additionally, the wearableextended reality appliance 5513 is configured to enable the user 5512 toview digital content presented via the physical display 5515 of thecomputing device 5514 through the wearable extended reality appliance5513. For illustration purposes, the wearable extended reality appliance5513 is depicted here as a pair of smart glasses, but as describedabove, wearable extended reality appliance 5513 may be any type ofhead-mounted device used for presenting an extended reality to user 5512when computing device 5514 is in the second display mode 5511.

As shown, the digital objects displayed to user 5512 when computingdevice 5514 is in the second display mode 5511 include real digitalobjects presented by physical display 5515 of computing device 5514 andvirtual digital objects presented by wearable extended reality appliance5513 in communication with computing device 5514. The real digitalobjects presented by physical display 5515 of computing device 5514 inthe second display mode 5511 include real digital object 5519B and realdigital object 5519C. Real digital object 5519B and real digital object5519C correspond in function and appearance to some of the plurality ofdigital objects presented to the user 5512 in the first display mode5510, as illustrated in FIG. 55A. The virtual digital objects presentedby wearable extended reality appliance 5513 include virtual digitalobject 5520 and virtual digital object 5521.

The virtual objects in the extended reality environment, as viewed fromthe perspective of user 5512, are depicted as two discrete virtualregions including virtual region 5522A and virtual region 5522B. Thevirtual regions 5522A and 5522B of the extended reality environment havebeen artificially imposed in this illustration to represent virtualcontent presented via wearable extended reality appliance 5513 from theperspective of user 5512. Virtual digital object 5520 is presented touser 5512 within virtual region 5522A away from and next to physicaldisplay 5515 such that virtual digital object 5520 appears to float at afixed location to the right of physical display 5515 from theperspective of user 5512. Virtual digital object 5521 is presented touser 5512 within virtual region 5522B in a region away from physicaldisplay 5515 corresponding to a surface to the left of computing device5514 such that virtual digital object 5521 appears to be positioned at afixed location on a surface to the left of physical display 5515 ofcomputing device 5514 from the perspective of user 5512.

The virtual digital content presented by wearable extended realityappliance 5513 may also include virtual cursor 5518B controllable viathe mouse 5517 and/or wearable extended reality appliance 5513. It is tobe understood that, in some embodiments, virtual cursor 5518B mayconstitute a virtual digital object, as described herein, when thecomputing device is in the second display mode 5511. However, a cursormay also be presented as a real digital object via the physical display5515 in some embodiments when the computing device is in the seconddisplay mode 5511. For example, the cursor may move anywhere withinphysical display 5515 as cursor 5518A and/or anywhere within theextended reality environment via wearable extended reality appliance5513 as virtual cursor 5518B. Additionally, the cursor may interact withany digital objects contained within the extended reality environmentincluding any real digital objects as cursor 5518A and/or as virtualcursor 5518B and/or any virtual digital objects as virtual cursor 5518B.

In the second display mode 5511, user 5512 may interact with any digitalobjects presented via wearable extended reality appliance 5513 and/orthe physical display 5515. In one example, virtual cursor 5518B may moveover a real digital object presented within physical display 5515 anddrag the real digital object out of physical display 5515 into theextended reality environment, for example to virtual region 5522A. Inanother example, virtual cursor 5518B may move anywhere within theextended reality environment, including virtual regions 5522A and 5522B,and may interact with, virtual digital object 5520 or virtual digitalobject 5521. In yet another example, virtual cursor 5518B may move onall available surfaces (e.g., virtual region 5522B or any otheridentifiable physical surface) or on selected surfaces in the extendedreality environment. Additionally, or alternatively, in the seconddisplay mode 5511, user 5512 may interact with any one of real digitalobjects 5519B or 5519C or virtual digital objects 5520 or 5521 usinghand gestures and/or eye gestures recognized by the wearable extendedreality appliance 5513 and/or any sensor (e.g., a camera) incommunication with computing device 5514.

In the second display mode, some of the plurality of digital objectspresented to the user in the first display mode may be presented to theuser via a physical display in communication with the computing device.The digital objects being presented via a physical display may besimilar to the real digital objects generated by the same computingdevice in the first display mode. Additionally, in the second displaymode, at least one other of the plurality of digital objects presentedto the user in the first display mode may be presented to the user viathe wearable extended reality appliance. The at least one other of theplurality of digital objects generated by the computing device in thesecond display mode and presented via the wearable extended realityappliance may include a virtual digital object or virtual digitalobjects consistent with the virtual digital objects described above. Insome examples, the at least one other of the plurality of digitalobjects may be presented via the wearable extended reality applianceover the physical display and/or away of the physical display. In someexamples, the at least one other of the plurality of digital objects maybe presented via the wearable extended reality appliance as a virtualdigital object in a fixed location relative to at least one particularphysical object, such as a desk, wall, digital device, or any physicalobject having at least one recognizable surface or boundary. Thelocation of the particular physical object itself may be fixed or notfixed relative to the entire space of the extended reality environment.

In one example, none of the at least one other of the plurality ofdigital objects is displayed via the physical screen. For example, thecomputing device operating in the second display mode may be configuredto present a plurality of digital objects such that none of at least onevirtual digital object presented via the wearable extended realityappliance may correspond to, and/or be displayed as, the real digitalobjects presented via the physical display in the second display mode.In another example, the at least one other of the plurality of digitalobjects is displayed via the physical screen while being displayed viathe wearable extended reality appliance. For example, the computingdevice operating in the second display mode may be configured to presenta plurality of digital objects such that at least one virtual digitalobject presented via the wearable extended reality appliance maycorrespond to, and/or be displayed as, at least one real digital objectpresented via the physical display in the second display mode. In yetanother example, a first digital object of the at least one other of theplurality of digital objects is displayed via the physical screen whilebeing displayed via the wearable extended reality appliance, and asecond digital object of the at least one other of the plurality ofdigital objects is not displayed via the physical screen. For example,the computing device operating in the second display mode may beconfigured to present a plurality of digital objects such that a firstdigital object may be presented as a real digital object via thephysical display while also being presented as a virtual digital objectvia the wearable extended reality appliance, and a second digital objectmay be presented as a virtual digital object via the wearable extendedreality appliance but not presented as a real digital object via thephysical display.

By way of a non-limiting example, one of the digital objects (realdigital object 5519A) illustrated in FIG. 55A is displayed via physicaldisplay 5515 as a weather widget in the first display mode 5510. Whencomputing device 5514 is operating in the second display mode 5511illustrated in FIG. 55B, real digital object 5519A is no longerpresented to user 5512 via physical display 5515. Rather, when computingdevice 5514 is operating in the second display mode 5511 illustrated inFIG. 55B, the digital object is presented as virtual digital object 5520to user 5512 (in virtual region 5522A) via wearable extended realityappliance 5513. In another example, when computing device 5514 isoperating in the second display mode 5511, a digital object (e.g., aweather widget) may be displayed via physical display 5515 while alsobeing presented to user 5512 (in virtual region 5522B) via wearableextended reality appliance 5513 such that a real digital object andvirtual digital object are simultaneously displayed to user 5512 withinthe extended reality environment. In yet another example, when computingdevice 5514 is operating in the second display mode 5511, a digitalobject (e.g., a weather widget) may be simultaneously displayed viaphysical display 5515 and in the extended reality environment viawearable extended reality appliance 5513 while another digital object(e.g., cursor 5518A), previously displayed in the first display mode5510, is displayed via wearable extended reality appliance 5513 (invirtual region 5522B) as virtual digital object 5518B, but not viaphysical display 5515.

In some embodiments, in the second display mode, the wearable extendedreality appliance may display at least one additional digital objectbeing excluded from display via the physical display in the firstdisplay mode. In a general sense, the “term excluded from display viathe physical display” may relate to any digital object that is virtuallypresented via the wearable extended reality appliance but not presented,or not configured to be presented, via the physical display incommunication with the computing device. As used herein, the term “oneadditional digital object” may relate to at least one digital objectthat is presented as a virtual digital object in the second display modevia the wearable extended reality appliance that is not displayed, orwas not previously displayed, as at least one real digital object in thefirst display mode via the physical display.

In one example, the wearable extended reality appliance may display atleast one virtual cursor located outside the physical boundaries of thephysical display. In another example, the wearable extended realityappliance may display at least one visual element that resides outsidethe physical boundaries of the physical display. For example, thewearable extended reality appliance may be configured to display one ormore of a virtual digital object for controlling at least one functionof the computing device, a virtual digital object which may activate ascript for causing an action associated with a particular digitalobject, and/or any other data representation or visual presentationdisplayed, or configured for display, via a wearable extended realityappliance. In one non-limiting example, a visual element that residesoutside the physical boundaries of the physical display may include atwo-dimensional (e.g., simplified) object, such as a clock on the wallor a virtual controller in communication with the computing device, ordisplayed as a three-dimensional life-like object, such as a plant on adesk.

By way of a non-limiting example, FIG. 55B illustrates virtual digitalobject 5521 (a volume controller) that is displayed via wearableextended reality appliance 5513 (in virtual region 5522B) when computingdevice 5514 is operating in the second display mode 5511. Virtualdigital object 5521 is not functionally related to any real digitalobject displayed via physical display 5515 in the second display mode5511 illustrated in FIG. 55B. Additionally, a digital objectcorresponding to virtual digital object 5521 is excluded from displayvia physical display 5515 in the first display mode 5510. For example,there is no digital object illustrated in FIG. 55A that is displayed viaphysical display 5515 that corresponds to virtual digital object 5521,as shown in FIG. 55B. In the example shown, virtual digital object 5521is configured to be interacted with only as a virtual digital object andnot configured to be displayed via physical display 5515 in the firstdisplay mode 5510.

In some embodiments, the at least one other of the plurality of digitalobjects has a first visual appearance when presented by the physicaldisplay in the first display mode and has a second visual appearancewhen presented by the wearable extended reality appliance in the seconddisplay mode. As used herein, the term “visual appearance” may relate tothe arrangement, layout, and/or overall presentation of a digital objectas displayed. For example, such an appearance may differ based onwhether presented on the physical display or the wearable extendedreality appliance. In one example, a particular digital object displayedvia the physical display may have a first visual appearance whenpresented via a physical display in the first display mode and a secondvisual appearance that is different from the first visual appearancewhen presented virtually in the second display mode via a wearableextended reality appliance. For example, a graphical user interface of aprogram for viewing and editing a document presented via a wearableextended reality appliance in the second display mode may be differentfrom the same program presented via a physical display in the firstdisplay mode. In another example, the visual appearance of a widget forchecking emails may appear in a simplified version when presented viathe physical display in the first display mode and may appear as anexpanded version with more functionality when presented via the wearableextended reality appliance when the computing device is operating in thesecond display mode. In another example, a real digital object displayedvia a physical display in the first display mode may have a visualappearance that is similar in at least one respect, or in all respects,to a virtual digital object displayed via the wearable extended realityappliance in the second display mode.

By way of a non-limiting example, FIG. 55B illustrates virtual digitalobject 5520 (relating to an application for checking the weather) in thesecond display mode 5511, which functionally corresponds to real digitalobject 5519A displayed in the first display mode 5510 of FIG. 55A. Thedigital object is displayed as an icon (e.g., real digital object 5519A)when presented via physical display 5515 in the first display mode 5510and is presented as an open application (e.g., virtual digital object5520) when displayed via wearable extended reality appliance 5513 (invirtual region 5522A) when the computing device is operating in thesecond display mode 5511. As shown, virtual digital object 5520 has afirst visual appearance when presented via the physical display 5515 inthe first display mode 5510 that is different from a second visualappearance when presented via the wearable extended reality appliance5513 in the second display mode 5511.

In some embodiments, the location of a particular digital object maydepend upon the location of some particular physical object. In otherembodiments, the location of a particular digital object may not dependupon the location of some particular physical object. In someembodiments, when the computing device is operating in the first displaymode, a location of a particular digital object of the at least oneother of the plurality of digital objects is independent of a locationof a particular physical object. The term “independent of a location ofa particular physical object” may refer to a particular digital object'sabsence of spatial dependence on or relationship to a location of aparticular physical object at a particular time and/or in a particulardisplay mode. For example, a location of a particular digital object,such as a real digital object, presented to the user via the physicaldisplay may not depend on, or otherwise rely upon, the location of somephysical object outside of the physical display. In one non-limitingexample, the location of a real digital object (e.g., a widget)displayed via a physical display of the computing device in the firstdisplay mode may remain within the same discrete subset of space of thephysical display, or may otherwise remain unaffected, if a distancebetween a particular physical object (e.g., a chair, table, or plant)changes relative to some point of reference, such as the physicaldisplay.

In some embodiments, when the computing device is operating in thesecond display mode, the location of the particular digital objectdepends on the location of the particular physical object. The term“depends on the location of the particular physical object” may refer toa particular digital object's spatial dependence on or relationship to alocation of a particular physical object at a particular time and/or ina particular display mode. In some embodiments, a location of aparticular digital object, such as a virtual digital object, presentedto the user via the wearable extended reality appliance in the seconddisplay mode may depend on, or otherwise rely upon, the location of somephysical object outside of the physical display. In some examples, theparticular virtual digital object may be docked to or near theparticular physical object. For example, when a virtual digital objectis docked to a physical object, the virtual digital object may stay inthe same location as the physical object and may move with the physicalobject. Additionally, or alternatively, when a virtual digital object isdocked near a physical object, the virtual digital object may stay at alocation that is within a fixed distance or range relative to thephysical object and may move with the physical object such that thedigital object remains within a fixed distance or range relative to thephysical object. When a virtual digital object is docked to a physicalobject, at least one point of the virtual digital object may stay fixedrelative to at least one point on the physical object such that aparticular position and/or angle of the virtual digital object maychange with a position and/or angle of the physical object.

In another example, the particular virtual digital object may beconfigured to move to and/or from the particular physical object. Forexample, a virtual digital object may be configured to move within theextended reality environment from a location that is a first distance(e.g., longer distance) from the particular physical object to alocation that is a second distance (e.g., shorter distance) from theparticular physical object, and vice versa. Additionally, oralternatively, a real digital object may be configured to move out ofdisplay via the physical display and onto display via the wearableextended reality appliance within the extended reality environment(e.g., as a virtual digital object) toward the particular physicalobject, and vice versa.

In another example, the particular virtual digital object may be locatedso that it does not hide, or otherwise obstruct, the particular physicalobject. In yet another example, the particular virtual digital objectmay be located so that it is not hidden, or otherwise obstructed, by theparticular physical object. For example, the location of a particularvirtual digital object (e.g., a widget for a computer application) maybe configured to move relative to a particular physical object (e.g.,any physical object within the extended reality environment such as adesk, chair, or peripheral device to the computing device) such that thevirtual digital object does not obstruct the user's view of the physicalobject and/or the user's view of the virtual digital object. If theparticular digital object is at a location that would obstruct theuser's view of the particular virtual digital object, the virtualdigital object may be configured to move to a new location relative tothe physical object as to remain visible to the user. Additionally, oralternatively, if the particular virtual digital object is at a locationthat would obstruct the user's view of the particular physical digitalobject, the virtual digital object may be configured to move to a newlocation relative to the physical object as to not block the user's viewof the physical object.

By way of a non-limiting example, FIG. 56 is a schematic illustration ofan example of a plurality of digital objects presented to a user in asecond display mode, consistent with some embodiments of the presentdisclosure. FIG. 56 illustrates a computing device operating in a seconddisplay mode, the computing device including physical display 5614 andkeyboard 5611. The physical display 5614 of the computing device isconfigured to display digital content, such as a plurality of realdigital objects 5615, to user 5612. For illustration purposes, physicaldisplay 5614 of the computing device is depicted here as a computermonitor configured to display digital content to the user 5612. Wearableextended reality appliance 5613 is in wireless communication with thecomputing device and is configured to display virtual digital content,such as virtual digital objects, to user 5612 when the computing deviceis in the second display mode. The virtual digital content displayed touser 5612 includes virtual digital objects 5616, virtual digital object5618, and virtual digital object 5619. When the computing device isoperating in the second display mode, the location of virtual digitalobjects 5616, virtual digital object 5618, and virtual digital object5619 depend on the location of particular physical objects.

As shown, the location of virtual digital objects 5616 depend on thelocation of physical display 5614. Virtual digital objects 5616 arecontained in a virtual region that is locked to physical display 5614 ata distance away from physical display 5614. Virtual digital objects 5616are also at a fixed position within the virtual region that is locked tophysical display 5614. In this example, the physical display 5614 actsas a physical object. In another example, virtual digital objects 5616may be locked at a distance from at least one of the real digitalobjects 5615 displayed via the physical display 5614. As anotherexample, the location of virtual digital object 5618 is docked on top ofkeyboard 5611 and depends on the location of keyboard 5611. The locationof virtual digital object 5618 depends on the location of keyboard 5611at least because one point of virtual digital object 5618 is fixedrelative to at least one point on keyboard 5611 such that a location ofvirtual digital object 5618 will stay in the same location as keyboard5611 and may move with keyboard 5611.

As another example, when the computing device is operating in the seconddisplay mode, the location of virtual digital object 5619 depends on thelocation of physical object 5617. For example, virtual digital object5619 is located at a set distance from physical object 5617 so that itdoes not hide the physical object 5617 from user 5612. In this example,physical object 5617 is a mobile communication device; however, asdiscussed above, it is to be understood that a physical object mayinclude any other physical object having at least one recognizablesurface or boundary. In one example, virtual digital object 5619 may beconfigured to move with, move to, and/or move from physical object 5617.In another example, the position of virtual digital object 5619 relativeto physical object 5617 may change in response to an action of physicalobject 5617 (e.g., receiving a text message).

Some embodiments involve determining a usage status of the wearableextended reality appliance. The term “usage status” may relate to astate of use, condition for use, and/or suitability for use of thewearable extended reality appliance described above at any point intime. In some embodiments, a usage status of the wearable extendedreality appliance may include a power status of the wearable extendedreality appliance (e.g., on or off), an engagement status of the user ofthe wearable extended reality appliance (e.g., whether the user isinteracting with the digital content or has interacted with the digitalcontent recently), a connection status of the wearable extended realityappliance (e.g., connected or disconnected to a computing device), abattery status of the wearable extended reality appliance (e.g., fullbattery, partial battery, low battery, or no battery), a hardware and/orsoftware status of the wearable extended reality appliance (e.g.,operating normally or abnormally), and/or any identifiable measure, orcombination of measures, of the wearable extended reality appliance'sstate, condition, and/or suitability for use.

As used herein, the determination of the usage status of the wearableextended reality appliance may be based on information related to pastand/or present usage, operation, and/or general utilization of aparticular computing device, the wearable extended reality appliance,and/or another device used in connection with the particular computingdevice and/or the wearable extended reality appliance. The determinationof the usage status of the wearable extended reality appliance may occurat any given point of time (e.g., at startup of the computing deviceand/or the wearable extended reality appliance) or over any period oftime. Any form of data and/or input received by, processed by, and/orstored by at least one computing device that is related to the wearableextended reality appliance's state, condition, and/or suitability foruse may be utilized to determine a usage status of the wearable extendedreality appliance. In one example, the usage status of the wearableextended reality appliance may be determined based on at least one formof data stored in the computing device and/or the wearable extendedreality appliance relating to past usage of the wearable extendedreality appliance (e.g., predicted behavior and/or preferences of thewearer of the wearable extended reality appliance). For example, if theuser has a tendency to wear the wearable extended reality appliance whenthey are finished using the wearable extended reality appliance, a usagestatus of the wearable extended reality appliance may indicate that theuser is no longer interacting with the digital content after some amountof time based on data related to prior use and/or user preferences.

In another example, the usage status of the wearable extended realityappliance may be determined based on at least one input received by thecomputing device and/or the wearable extended reality appliance,indicating the wearable extended reality appliance is ready for use. Forexample, a webcam may capture image data indicating the user is wearingthe wearable extended reality appliance. Additionally, or alternatively,a sensor on the computing device or the wearable extended realityappliance may indicate that the user is wearing the wearable extendedreality appliance or that the wearable extended reality appliance hassufficient battery for use within the extended reality environment.

In one embodiment, the usage status of the wearable extended realityappliance is determined based on data indicating when the wearableextended reality appliance is active. As used herein, the term “active”may refer to an activity status of the wearable extended realityappliance indicating that the wearable extended reality appliance and/orthe user of the wearable extended reality appliance is engaged or readyto engage or interact with the extended reality environment. In someembodiments, data indicating when the wearable extended realityappliance is active may relate to any information used by the computingdevice and/or wearable extended reality appliance to determine oranalyze the activity of the wearable extended reality appliance and/orthe user to determine if the wearable extended reality appliance isready for use, and/or the user is ready to use the wearable extendedreality appliance, in the extended reality environment.

In one example, the wearable extended reality appliance may be activewhen the power status of the wearable extended reality appliance is“on.” The wearable extended reality appliance may be turned on bypressing a power button on the wearable extended reality appliance suchthat power is delivered to individual components of the wearableextended reality appliance. In another example, the wearable extendedreality appliance may be active when the user of the wearable extendedreality appliance is engaged with the extended reality environment orwhen digital content is ready for display within the extended realityenvironment via a wearable extended reality appliance. The user of thewearable extended reality appliance may be engaged with the wearableextended reality appliance when it is turned on, when the user is readyto use the wearable extended reality appliance (e.g., computing deviceis on and the user is near the physical display), and/or when the useris currently using the wearable extended reality appliance. In yetanother example, the wearable extended reality appliance may be activewhen the wearable extended reality appliance is connected to, orotherwise in communication with, a computing device. Alternatively, thewearable extended reality appliance may be inactive when the wearableextended reality appliance is disconnected from the computing device

In another embodiment, the usage status of the wearable extended realityappliance is determined based on data indicating when the wearableextended reality appliance is physically connected through a wire to aport of the computing device. As used herein, the term “physicallyconnected” may be used to refer to a connection status of the wearableextended reality appliance in which the wearable extended realityappliance is attached via a wire to a computing device and/or to atleast one peripheral device of the computing device including akeyboard, a mouse, or a monitor. When the wearable extended realityappliance is physically connected to the computing device through a wireto a port (e.g., a port capable of facilitating the transmission of datarelated to the usage status of the wearable extended reality appliance)of the computing device, the wearable extended reality appliance may beconfigured to transmit information directly or indirectly to thecomputing device. In one example, the wearable extended realityappliance may be configured to charge and transmit information to thecomputing device simultaneously. In another example, the wearableextended reality appliance may be physically connected to a wiredkeyboard connectable to a computing device via at least one input of thecomputing device. In another example, the wearable extended realityappliance may be physically connected to a wireless keyboard incommunication with the computing device. In yet another example, thewearable extended reality appliance may be physically connected to aphysical display in communication with the computing device.

In another embodiment, the usage status of the wearable extended realityappliance is determined based on input from a sensor indicating when thewearable extended reality appliance is worn. As used herein, the term“sensor” may relate to any device in communication with the wearableextended reality appliance and/or the computing device configured todetect and/or measure a property associated with the user, the user'saction, user's environment, and/or a property associated with thewearable extended reality appliance. In one example, sensor data may bebased on information captured using one or more sensors of an inputdevice in communication with the computing device. In another example,sensor data may be based on information captured using one or moresensors of the extended reality appliance. In yet another example,sensor data may be based on information captured using a combination oneor more sensors of an input device in communication with the computingdevice and using one or more sensors of the extended reality appliance.

In some embodiments, the sensor may include one or more image sensors(e.g., configured to capture images and/or videos of a user of theappliance or of an environment of the user), one or more motion sensors(such as an accelerometer, a gyroscope, a magnetometer, etc.), one ormore positioning sensors (such as GPS, outdoor positioning sensor,indoor positioning sensor, etc.), one or more temperature sensors (e.g.,configured to measure the temperature of at least part of the applianceand/or of the environment), one or more contact sensors, one or moreproximity sensors (e.g., configured to detect whether the appliance iscurrently worn), one or more electrical impedance sensors (e.g.,configured to measure electrical impedance of the user), one or more eyetracking sensors, such as gaze detectors, optical trackers, electricpotential trackers (e.g., electrooculogram (EOG) sensors), video-basedeye-trackers, infra-red/near infra-red sensors, passive light sensors,or any other technology capable of determining a usage status of thewearable extended reality appliance.

The computing device may use input data (e.g., stimulus, response,command, and/or instruction targeted to a processing device) from atleast one sensor to determine the usage status of the wearable extendedreality appliance. For example, an input may be received by the at leastone processor via an input interface (e.g., input interface 430 of FIG.4 and/or input interface 330 of FIG. 3 ), by a sensor associated withthe wearable extended reality appliance (e.g., sensor interface 470 or370), by a different computing device communicatively coupled to thewearable extended reality appliance (e.g., mobile device 206 and/orremote processing unit 208 of FIG. 1 ), or any other source of input,for example, a camera (e.g., as gesture input or input relating to theusage of the wearable extended reality appliance).

As used herein, “input from a sensor indicating when the wearableextended reality appliance is worn” may relate to any sensor dataindicating that the wearable extended reality appliance is currently on,donned by, or a part of a user as a wearable electronic device such thatthe wearable extended reality appliance is capable of presenting anextended reality to the user. In one example, a proximity sensor, orcombination of proximity sensors, connected to a physical display mayprovide sensor data indicating a presence of the wearable extendedreality appliance in proximity to the physical display. For example,data from proximity sensors, or a combination of proximity sensors maybe used to determine that the wearable extended reality appliance is ina position relative to the computing device or a peripheral device thatis indicative of the wearable extended reality appliance being worn. Inanother example, an electrical impedance sensor and/or a motion sensorincluded in the wearable extended reality appliance may provide sensordata indicating the user is ready to engage or interact with theextended reality environment via the wearable extended realityappliance.

In another embodiment, the usage status of the wearable extended realityappliance is determined based on image data captured using an imagesensor. The term “image sensor” may include any instrument or group ofinstruments capable of converting rays of light (e.g., photons) intoelectrical signals. Examples of image sensors include CCD and CMOSarrays. Other types of image sensors include Lidar and radar sensors. Insome examples, the image sensor may be included in the extended realityappliance, in the computing device, in an input device, and/or in anenvironment of the user. In one example, the image sensor may be in oron a laptop or computer monitor in communication with the computingdevice such as an integrated, built-in, or standalone webcam. In anotherexample, the image sensor may be a part of the extended realityappliance.

As used herein, the term “image data” may relate to any data captured byone or more image sensors and may be understood as described earlier. Atleast one processor may be configured to determine the usage status ofthe wearable extended reality appliance based on image data from anycombination of signals emitted and/or reflected off physical objects inthe extended reality environment, data stored in memory (e.g., for thelocation of stationary objects), predicted behavior and/or preferencesof the wearer of the wearable extended reality appliance, ambientconditions (e.g., light, sound, dust), and any other criterion fordetermining a relative position of the wearable extended realityappliance and/or physical objects in the extended reality environment.The signals may include any combination of image data and/or IR signalsdetected by a camera (e.g., image sensor 472 of FIG. 4 ), position,location, and orientation data acquired by an IMU and/or GPS unit (e.g.,motion sensor 473), ultrasound, radio (e.g., Wi-Fi, Bluetooth, Zigbee,RFID) detected via suitable sensors (e.g., other sensors 475). In oneexample, the image data may indicate the proximity of the user andwhether the user is wearing the wearable extended reality appliance. Inanother example, the image data may indicate the position of the userrelative to the wearable extended reality appliance. In another example,the image data may indicate the position of the user and/or the wearableextended reality appliance relative to the physical display.

In some examples, the image data captured using the image sensor may beanalyzed to determine the usage status of the wearable extended realityappliance. For example, a machine learning model (such as aclassification model) may be trained using training examples todetermine usage statuses of wearable extended reality appliances fromimages and/or videos. An example of such training example may include asample image and/or sample video associated with a sample wearableextended reality appliance, together with a label indicating the usagestatus of the sample wearable extended reality appliance. The trainedmachine learning model may be used to analyze the image data capturedusing the image sensor and determine the usage status of the wearableextended reality appliance. In some examples, at least part of the imagedata may be analyzed to calculate a convolution of the at least part ofthe image data and thereby obtain a result value of the calculatedconvolution. Further, in response to the result value of the calculatedconvolution being a first value, one usage status of the wearableextended reality appliance may be determined, and in response to theresult value of the calculated convolution being a second value, anotherusage status of the wearable extended reality appliance may bedetermined. In some examples, the image data may be analyzed todetermine a type of environment of the wearable extended realityappliance, for example using scene recognition algorithms, and the usagestatus of the wearable extended reality appliance may be determinedbased on the type of the environment of the wearable extended realityappliance. In some examples, the image data may be analyzed to detectobjects in the environment of the wearable extended reality appliance,for example using object detection algorithms, and the usage status ofthe wearable extended reality appliance may be determined based on theobjects in the environment of the wearable extended reality appliance.In some examples, the image data may be analyzed to detect activities inthe environment of the wearable extended reality appliance, for exampleusing event detection algorithms, and the usage status of the wearableextended reality appliance may be determined based on the activities inthe environment of the wearable extended reality appliance.

In another embodiment, the usage status of the wearable extended realityappliance is determined based on data indicating when a communicationchannel is established between the computing device and the wearableextended reality appliance. The term “communication channel” includesany single or group of wired or wireless pathways or other medium overwhich data or information exchanges may occur. Such channels may permitthe transport of data and/or information signals from one or moretransmitters to one or more receivers. In one example, a wiredtransmission medium may relate to a wired communication channelconfigured to transport data and/or information between the computingdevice and the wearable extended reality appliance. In another example,a wireless transmission medium may relate to a wireless communicationchannel configured to transport data and/or information from between thecomputing device and the wearable extended reality appliance via atleast one wireless network. For example, one or more components of thewearable extended reality appliance and/or computing device maycommunicate directly through a dedicated communication network includingBLUETOOTH™, BLUETOOTH LE™ (BLE), Wi-Fi, near field communications (NFC),and/or any other suitable communication methods that provide a mediumfor exchanging data and/or information between the wearable extendedreality appliance and the computing device.

As used herein, a communication channel is “established” between thecomputing device and the wearable extended reality appliance when thecomputing device is connected to the wearable extended reality applianceand able to transmit information to and/or receive information from thewearable extended reality appliance via the communication channel. Theterm “data indicating when a communication channel is established” mayrelate to any information used by the computing device and/or wearableextended reality appliance to determine the connection status of thewearable extended reality appliance (e.g., connected or disconnected tothe computing device). In one example, a connection status of thewearable extended reality appliance may indicate that a communicationchannel is established (e.g., exchange of data or information ispossible) between the computing device and the wearable extended realityappliance. In another example, a connection status of the wearableextended reality appliance may indicate that a communication channel isnot established (e.g., exchange of data or information is not possible)between the computing device and the wearable extended realityappliance.

In another embodiment, the usage status of the wearable extended realityappliance is determined based on data indicative of a battery status ofthe wearable extended reality appliance. As used herein, the “batterystatus” may refer to the amount of battery life remaining in thewearable extended reality appliance and/or the time it will take todischarge and/or charge the wearable extended reality appliance ifconnected to a power source. In some examples, data indicative of abattery status of the wearable extended reality appliance may relate toany information used by the computing device and/or wearable extendedreality appliance to determine the battery status of the wearableextended reality appliance and/or determine if the wearable extendedreality appliance is suitable for use in the extended realityenvironment. In one example, when the battery status indicates thebattery is full battery or partial battery, the wearable extendedreality appliance may be suitable for use in the extended realityenvironment. In another example, when the battery status indicates thebattery is low battery or no battery, the wearable extended realityappliance may not be suitable for use in the extended realityenvironment.

Some embodiments involve selecting a display mode based on the usagestatus of the wearable extended reality appliance. The term “selecting adisplay mode” may refer to a picking or choosing a display mode fordisplaying digital objects from among a plurality of display modes, asdisplay modes are described earlier. For example, the computing devicemay select a first display mode or a second display mode in view of thedetermined usage status (e.g., a first usage status or a second usagestatus). In some embodiments, the usage status of a wearable extendedreality appliance may inform the determination of the display mode inwhich certain digital objects are displayed for presentation to a uservia a physical display of a computing device and/or a wearable extendedreality appliance. In one example, a first usage status of the wearableextended reality appliance may inform the computing device that thewearable extended reality appliance is unsuitable for use in theextended reality environment at or for a particular time. When the usagestatus of the wearable extended reality appliance indicates that thewearable extended reality appliance is in a first usage status, thecomputing device may select a first display mode in which digitalobjects are displayed via physical display.

By way of a non-limiting example, turning to FIG. 55A, user 5512 isshown not wearing a wearable extended reality appliance. Because user5512 is not wearing a wearable extended reality appliance, data from atleast one sensor of a wearable extended reality appliance and/or atleast one image sensor of computing device 5514 indicates the wearableextended appliance is not being worn by user 5512. In view of dataindicating the wearable extended appliance is not being worn by user5512, a processor of computing device 5514 determines the usage statusof the wearable extended reality appliance is a first usage status. Whenthe usage status of a wearable extended reality appliance is a firstusage status, the at least one processor selects the first display mode5510 for displaying real digital objects 5519A to 5519C to user 5512 viaphysical display 5515 of computing device 5514.

In another example, a second usage status of the wearable extendedreality appliance may inform the computing device that the wearableextended reality appliance is suitable for use in an extended realityenvironment at or for a particular time. When the usage status of thewearable extended reality appliance indicates that the wearable extendedreality appliance is in a second usage status, the computing device mayselect a second display mode in which virtual digital objects aredisplayed via the wearable extended reality appliance. In someembodiments, the computing device may be configured to switch betweendifferent selected display modes in real time or near real time inresponse to a change in usage status of the wearable extended realityappliance.

By way of a non-limiting example, turning to FIG. 55B, user 5512 isshown wearing wearable extended reality appliance 5513. Because user5512 is wearing wearable extended reality appliance 5513, data from atleast one sensor of the wearable extended reality appliance 5513 and/orat least one image sensor of computing device 5514 indicates wearableextended reality appliance 5513 is being worn by user 5512.Additionally, wearable extended reality appliance 5513 is shown to bephysically connected to computing device 5514 and a communicationchannel is established between computing device 5514 and wearableextended reality appliance 5513. Because wearable extended realityappliance 5513 is physically connected and in communication with thecomputing device 5514, data received by the computing device indicateswearable extended reality appliance 5513 is in condition for use.

In view of data indicating the wearable extended reality appliance 5513being worn by user 5512, is connected to computing device 5514, and isin communication with computing device 5514, a processor of computingdevice 5514 determines the usage status of the wearable extended realityappliance is a second usage status. When the usage status of wearableextended reality appliance 5513 is a second usage status, the at leastone processor selects the second display mode 5511 for displaying realdigital objects 5519B and 5519C to user 5512 via physical display 5515and virtual digital objects 5520 and 5521 via wearable extended realityappliance 5513.

Some embodiments involve, in response to the display mode selection,outputting for presentation the plurality of digital objects in a mannerconsistent with the selected display mode. The term “outputting forpresentation” relates to a transmission of signals to cause digitalcontent or virtual digital content to be presented for viewing by a uservia a physical display or via a wearable extended reality appliance. Insome embodiments, the outputting of digital objects for presentation mayoccur after a display mode has been selected and/or after a display modehas changed. As used herein. “a manner consistent with the selecteddisplay mode” may refer to the way in which real digital objects and/orvirtual digital objects may be displayed, shown, or caused to appear forview by a user according to the method specified by the selected displaymode. For example, when the first display mode is selected, thecomputing device may output the plurality of digital objects for displayvia the physical display. In another example, when the second displaymode is selected, the computing device may output some of the pluralityof digital objects for display via the physical display and at least oneother of the plurality of digital objects for display via the wearableextended reality appliance.

By way of a non-limiting example, turning to FIG. 55A, when the firstdisplay mode 5510 is selected, at least one processor of the computingdevice 5514 outputs for presentation the plurality of digital objects ina manner consistent with the first display mode 5510. Here, theplurality of digital objects includes real digital objects 5519A to5519C to be displayed to user 5512 via physical display 5515 ofcomputing device 5514. Turning to FIG. 55B, when the second display mode5511 is selected, at least one processor of the computing device 5514outputs for presentation the plurality of digital objects in a mannerconsistent with the second display mode 5511. Here, the plurality ofdigital objects includes real digital objects 5519B and 5519C to bedisplayed to user 5512 via physical display 5515 and virtual digitalobjects 5520 and 5521 to be displayed to user 5512 via wearable extendedreality appliance 5513.

In some embodiments, when the selected display mode is the seconddisplay mode, outputting for presentation the plurality of digitalobjects includes causing the at least one other of the plurality ofdigital objects to be displayed via the wearable extended realityappliance while the some of the plurality of digital objects areconcurrently displayed via the physical display. “At least one other” ofthe plurality of digital objects are generated by the computing devicein the second display mode and presented via the wearable extendedreality appliance. The at least one other of the plurality of digitalobjects includes a virtual digital object or virtual digital objectsconsistent with the virtual digital objects described above. As usedherein, the term “concurrently displayed” refers to the simultaneousdisplay of real digital objects to the user via the physical display andvirtual digital objects to the user via the wearable extended realityappliance. For example, in the second display mode, the at least oneother of the plurality of digital objects (i.e., at least one virtualdigital object) may be displayed to the user via the wearable extendedreality appliance while some of the plurality of digital objects (i.e.,some of the real digital objects) are displayed to the user via thephysical display at the same time.

By way of a non-limiting example, turning to FIG. 55B, when the seconddisplay mode 5511 is selected, at least one processor of computingdevice 5514 outputs for presentation the plurality of digital objects ina manner consistent with the second display mode 5511. Here, theplurality of digital objects includes real digital objects 5519B and5519C to be displayed to user 5512 via physical display 5515 and virtualdigital objects 5520 and 5521 to be displayed to user 5512 via wearableextended reality appliance 5513. At least one other of the plurality ofdigital objects (e.g., virtual digital object 5520) is displayed viawearable extended reality appliance 5513 while the some of the pluralityof digital objects (e.g., real digital object 5519B and real digitalobject 5519C) are concurrently displayed via the physical display 5515.

In another embodiment, when the selected display mode is the seconddisplay mode, outputting for presentation the plurality of digitalobjects includes presenting at least one digital object concurrently viathe wearable extended reality appliance and via the physical display.For example, in the second display mode, at least one digital object maybe displayed to the user via the wearable extended reality appliance andvia the physical display at the same time. The at least one digitalobject may be presented to the user as a virtual digital object via thewearable extended reality appliance and concurrently presented to theuser as a real digital object via the physical display.

By way of a non-limiting example, turning to FIG. 56 when the seconddisplay mode is selected, at least one processor of the computing deviceoutputs for presentation the plurality of digital objects including realdigital objects 5615 to be displayed to user 5612 via physical display5614 and virtual digital objects 5616, virtual digital object 5618, andvirtual digital object 5619 to be displayed to user 5612 via wearableextended reality appliance 5613. Here, a digital object for controllingcomputer settings is concurrently displayed via wearable extendedreality appliance 5513 (e.g., as virtual digital object 5618) and viathe physical display 5614 (e.g., as one of real digital objects 5615).

Some embodiments involve determining to display, in the second displaymode, the at least one other of the plurality of digital objects via thewearable extended reality appliance. As used herein, “determining todisplay in the second display mode” may refer to a decision or causationto present particular digital objects, or groups of digital objects, tothe user in the second display mode. In some embodiments, determiningwhether to display in the second display mode may be based on dataand/or information related to past and/or present usage of the wearableextended reality appliance and/or any other peripheral device incommunication with the computing device. A peripheral device may includeany input and/or output device or devices directly or indirectlyconnected to, or otherwise in communication with, a computing deviceand/or the extended reality appliance. In some embodiments, determiningto display in the second display mode may be based on data and/orinformation related to a specific digital object and/or groups ofdigital objects configured to be presented to a user. For example,information related to the usage, form, and/or function of a specificdigital object, or groups of digital objects, may be utilized todetermine which digital object, or groups of digital objects, are to bedisplayed for presentation via a physical display of a computing deviceand which of the digital objects are to be displayed via the wearableextended reality appliance.

In one embodiment, the determining to display, in the second displaymode, the at least one other of the plurality of digital objects via thewearable extended reality appliance is based on user input. The term“user input” may refer to any information and/or data that is sent bythe user and received by the computing device for processing. The usermay transmit input data and/or information from a variety of inputdevices, for example, a keyboard, a mouse, a touch pad, a touch screen,one or more buttons, a joystick, a microphone, an image sensor, and anyother device configured to detect physical or virtual input. Thereceived input may be in the form of at least one of text, sounds,speech, hand gestures, body gestures, tactile information, and any othertype of physically or virtually input generated by the user. In someembodiments, the input received from the user may be used to determinewhich of the digital objects to display via the physical display andwhich to display via the wearable extended reality appliance. In someembodiments, the user's input or inputs may be detected and convertedinto virtual interactions with the extended reality environment therebyenabling the user to select, or otherwise interact with, digital objectswithin the extended reality environment. For example, a user may dragand drop a digital object from a real environment to an extended realityenvironment, and vice versa.

In one example, when the computing device is in the second display mode,the user may select, or otherwise interact with, at least one digitalobject via hand gestures. For example, the user may drag a real digitalobject displayed via a physical display from the physical display to thewearable extended reality appliance such that the digital object isdisplayed as a virtual digital object. In another example, when thecomputing device is in the second display mode, the user may select, orotherwise interact with, at least one digital object via eye gestures.For example, the user may drag a virtual digital object displayed viathe wearable extended reality appliance from the extended realityenvironment to the physical display such that the digital object isdisplayed as a real digital object. In yet another example, when thecomputing device is in the second display mode, the user may select, orotherwise interact with, at least one digital object via a cursor byusing a computer mouse. For example, the user may drag and drop realdigital objects and/or virtual digital objects between the physicaldisplay and the wearable extended reality appliance within the extendedreality environment.

In one embodiment, the determining to display, in the second displaymode, the at least one other of the plurality of digital objects via thewearable extended reality appliance is based on a type of input deviceconnected to the computing device. As used herein, the term “inputdevice” may refer to any device, or combination of devices, configuredto provide information and/or data to the computing device forprocessing. The input device may include any of the above-describedinput devices configured to detect a physical and/or digital input. Asused herein, the term “type of input device” may refer to the categoryof input device, or input devices, in communication with a computingdevice. In some embodiments, the category of input device may relate tothe use of certain input devices with a particular computing device(e.g., a wireless keyboard vs. a wired keyboard) and/or in a particularenvironment (e.g., at a user's home office space vs. work office space).

In some embodiments, the category of input device, or input devices, mayinform which digital object or digital objects are displayed via thewearable extended reality appliance. In one embodiment, input devicesused in conjunction with one particular computing device (e.g., adesktop computer) may relate to one category of input devices (e.g.,devices used at a user's work) and input devices used in conjunctionwith another particular computing device (e.g., a laptop) may relate toanother category of input devices (e.g., devices used at a user's home).For example, an input device (e.g., a wired mouse) used in conjunctionwith the desktop computer may cause at least one particular digitalobject to be presented to the user via the wearable extended realityappliance, and an input device (e.g., an integrated trackpad) used inconjunction with the laptop computer may cause another particulardigital object to be presented to the user via the wearable extendedreality appliance. In another embodiment, at least one input device usedwith a particular computing device's operation in one setting (e.g., theuser's home network) may relate to one category of input devices and atleast another input device used with the particular computing device'soperation in another setting (e.g., the user's office network) mayrelate to another category of input devices. For example, when a user isusing a laptop on their home network, at least one particular digitalobject may be presented to the user via the wearable extended realityappliance, and when a user is using the laptop on their work network, atleast another particular digital object may be presented to the user viathe wearable extended reality appliance. In another example, when a useris using a laptop on their home network, at least one particular digitalobject having a first appearance may be presented to the user via thewearable extended reality appliance, and when a user is using the laptopon their work network, the at least one particular digital object havinga second appearance may be presented to the user via the wearableextended reality appliance.

Additionally, or alternatively, the type of input device may refer tothe unique characteristics of a particular input device in communicationwith a computing device. In some embodiments, the unique characteristicsof an input device, or input devices, in communication with thecomputing device may inform which digital object or digital objects aredisplayed via the wearable extended reality appliance. In one example,in the second display mode, when a first input device (e.g., a wiredkeyboard) is connected to the computing device, at least one of a firstgroup of digital objects may be presented to the user via the wearableextended reality appliance. In another example, in the second displaymode, when a second input device (e.g., a wireless keyboard) isconnected to the computing device, at least one of a second group ofdigital objects may be presented to the user via the wearable extendedreality appliance. In yet another example, in the second display mode,when the second input device and a third input device (e.g., a wirelessmouse) are connected, at least one of a second group of digital objectsand/or a third group of digital objects may be presented to the user viathe wearable extended reality appliance.

In one embodiment, determining to display, in the second display mode,the at least one other of the plurality of digital objects via thewearable extended reality appliance is based on past user actions. Theterm “past user actions” may relate to stored information and/or datacorresponding to a user's former interactions with the extended realityenvironment. In some embodiments, a user's past interactions with theextended reality environment may include a user's prior virtual digitalobject selection, a user's interactions with digital objects inparticular settings and/or at particular times, privacy levelsassociated with different virtual digital objects, the relationshipbetween virtual digital objects and physical objects, the relationshipbetween virtual digital objects and real digital objects, the user'spreferences, the user's past behavior, and any other information and/ordata associated with a user's past usage within an extended realityenvironment.

In one example, a user's past actions may relate to the last time theuser docked the at least one digital object to a particular physicalobject. Because of the past action concerning the docking of the atleast one digital object, it may be determined to display the digitalobject via a wearable extended reality appliance (e.g., docked to theparticular physical object) and/or to display the digital object via aphysical display. In another example, a user's past actions may relateto the last digital object a user was interacting with in a particularsetting. Because of the past action concerning the last digital object auser was interacting with in a particular setting, it may be determinedto display the digital object via a wearable extended reality appliance(e.g., virtually displaying a digital object relative to one physicalobject within a work setting) and/or to display the digital object via aphysical display (e.g., not virtually displaying a digital objectrelative to one physical object within a home setting and onlydisplaying the digital object via a physical display).

In another example, a user's past actions may relate to the last digitalobject a user was interacting with at a particular time. Because of thepast action concerning the last digital object a user was interactingwith at a particular time, it may be determined to display the digitalobject via a wearable extended reality appliance (e.g., if a digitalobject is routinely opened for display via a wearable extended realityappliance at a first time of the day) and/or to display the digitalobject via a physical display (e.g., if the same digital object isroutinely opened for display via a physical display at a second time ofthe day). In another example, a user's past actions may relate todisplay preferences corresponding to certain virtual digital objects.Because of the past action concerning display preferences correspondingto certain virtual digital objects, it may be determined to display thecertain digital object via a wearable extended reality appliance and/ora physical display (e.g., when they are commonly opened when otherdigital objects are open).

In one embodiment, determining to display, in the second display modethe at least one other of the plurality of digital objects via thewearable extended reality appliance is based on at least one predefinedrule. The term “predefined rule” includes any predetermined conditionthat serves as a trigger for object display. In some embodiments, apredefined rule may delineate parameters for what digital objects todisplay, or not display, as virtual digital objects via the wearableextended reality appliance when the computing device is in the seconddisplay mode. In some embodiments, at least one predefined rule may betriggered at or for a particular time in response to information and/ordata corresponding to at least one action, option, and/or environmentthat may be distinguishable from some other action, option, and/orenvironment. For example, at least one predefined rule may be definedsuch that a weather widget must be presented virtually. In anotherexample, at least one predefine rule may be defined such that a textediting application must be presented virtually to allow user to edit ina magnified view.

In some embodiments, multiple predefined rules may be triggeredsimultaneously at the same time or over a period of time. For example,at least one predefined rule may be defined such that when mathematicalequations are present on a physical display, a calculator and a notepadare presented virtually in an extended reality environment via thewearable extended reality appliance while other virtual digital objectsare caused to disappear from the extended reality environment. Inanother embodiment, a predefined rule may call for a particular group ofdigital objects to be displayed, or not displayed, as virtual digitalobjects, in response to user input. For example, at least one predefinedrule may be defined such that when a user opens an application using amouse as an input, the application is displayed to the user via thephysical display, and when a user opens an application using handgestures as an input, the application is displayed to the user via thewearable extended reality appliance.

In some embodiments, a predefined rule may call for a particular groupof digital objects to be displayed, or not displayed, as virtual digitalobjects, in response to certain operating conditions of the computingdevice and/or the wearable extended reality appliance. For example, atleast one predefined rule may be defined such that when the battery ofthe wearable extended reality appliance and/or the computing device islow (e.g., below 20%), a first subset of applications may be availablefor display via the wearable extended reality appliance, and when thebattery of the wearable extended reality appliance and/or the computingdevice is very low (e.g., below 10%), a second subset of applicationsthat is less than the first subset of applications may be available fordisplay via the wearable extended reality appliance. In yet anotherembodiment, a predefined rule may define the manner in which certainvirtual digital objects are presented to the user. For example, at leastone predefined rule may be defined such when a particular digitalobject, such as a clock widget, is displayed virtually, the clock widgetis displayed at a particular location on a user's wall.

Some embodiments involve identifying a change in the usage status of thewearable extended reality appliance. In one example, while the pluralityof digital objects are presented in a manner consistent with the firstdisplay mode, the at least one processor may perform operations foridentifying a change in the usage status of the wearable extendedreality appliance from a first usage status corresponding to the firstdisplay mode to a second usage status corresponding to the seconddisplay mode. In another example, while the plurality of digital objectsare presented in a manner consistent with the second display mode, theat least one processor may perform operations for identifying a changein the usage status of the wearable extended reality appliance.

As used herein, a “change in the usage status” of the wearable extendedreality appliance may relate to any identifiable adjustment,modification, revision, shift, transition, or variation in a state ofuse, condition for use, and/or suitability for use of the wearableextended reality appliance. For example, an identifiable change in theusage status may include a change from one particular usage status(e.g., a first usage status of a high battery level) of the wearableextended reality appliance to another usage status (e.g., a second usagestatus of a low battery level) or vice-versa that is inconsistent with acurrent display mode. In another example, an identifiable change in theusage status may relate to a change in the user's degree of interactionwith digital content (e.g., the user has not interacted with digitalcontent in the past 5 minutes). In another example, an identifiablechange in the usage status may relate to a change in the connectionstatus of the wearable extended reality appliance (e.g., connecting thewearable extended reality appliance to a computing device).

In one example, a change in the usage status of the wearable extendedreality appliance may relate to a change from the first usage statuscorresponding to the first display mode to a second usage statuscorresponding to the second display mode. In another example, a changein the usage status of the wearable extended reality appliance mayrelate to a change from a second usage status corresponding to thesecond display mode to a first usage status corresponding to the firstdisplay mode. In yet another example, a change in the usage status ofthe wearable extended reality appliance may relate to a change from asecond usage status corresponding to the second display mode to a thirdusage status corresponding to a display mode that is different from thefirst display mode and the second display mode.

As used herein, “identifying a change in the usage status” may includesensing, detecting, or receiving an indication that a usage status, aspreviously described, has changed. The identification may be conductedin a manner similar to the processes for determining a usage status ofthe wearable extended reality appliance described above. In oneembodiment, any identifiable measure, or combination of measures, of thewearable extended reality appliance's state, condition, and/orsuitability for use may be utilized to identify a change in the usagestatus of the wearable extended reality appliance. For example, a changein the usage status of the wearable extended reality appliance may bedetected in response to battery voltage, temperature of operatingcomponents, or ambient light and/or sound conditions, the location ofphysical objects, and/or any other factor related to the wearableextended reality appliances usage. In one embodiment, the change in theusage status of the wearable extended reality appliance may beidentified based on any form of data and/or information received by,processed by, and/or stored by at least one computing device that isrelated to a change in the usage of the wearable extended realityappliance's state, condition, and/or suitability for use. In anotherexample, the change in the usage status of the wearable extended realityappliance may be identified based on at least one input received by thecomputing device and/or the wearable extended reality applianceindicating the wearable extended reality appliance is appropriate foruse or no longer appropriate for use.

Some embodiments involve updating the display mode selection and/orrevising the presentation of the plurality of digital objects inresponse to an identified change of the usage status of the wearableextended reality appliance. “Updating the display mode selection” mayrelate to a change to the display mode in response to an identifiedchange to the usage status of the wearable extended reality appliance.In some embodiments, the particular display mode in which the computingdevice was operating prior to the identified change in usage status ofthe wearable extended reality appliance may be updated to anotherdisplay mode that is consistent with the present usage status. In someembodiments, when the plurality of digital objects are presented in amanner consistent with the second display mode and a change in the usagestatus of the wearable extended reality appliance is identified, inresponse to the identified change in the usage status, the at least oneprocessor may perform operations for updating the display mode selectionfrom the second display mode to the first display mode. For example, inresponse to an identified change from the second usage statuscorresponding to the second display mode to a first usage status, thedisplay mode may be updated from the second display mode to the firstdisplay mode. In another example, in response to an identified changefrom the first usage status corresponding to the first display mode to asecond usage status, the display mode may be updated from the firstdisplay mode to the second display mode.

By way of a non-limiting example, referring back to FIGS. 55A and 55B,computing device 5514 is configured to analyze input signals indicatingwhen wearable extended reality appliance 5513 is in a first usage statusor a second usage status. When the wearable extended reality applianceis in a first usage status (e.g., not being worn by user 5512), asillustrated in FIG. 55A, the plurality of digital objects (e.g., realdigital objects 5519A to 5519C and cursor 5518A) are presented to user5512 via physical display 5515 in a manner consistent with the firstdisplay mode 5510. When wearable extended reality appliance 5513 is in asecond usage status (e.g., worn by user 5512), as illustrated in FIG.55B, the plurality of digital objects (e.g., real digital objects 5519Band 5519C, virtual cursor 5518B, virtual digital object 5520, andvirtual digital object 5521) are presented to user 5512 in a mannerconsistent with the second display mode 5511. As shown, real digitalobjects 5519B and 5519C are presented via the physical display 5515.Virtual cursor 5518B, virtual digital object 5520, and virtual digitalobject 5521 are presented via wearable extended reality appliance 5513.

When computing device 5514 identifies the usage status of the wearableextended reality appliance 5513 has changed from a first usage status(as shown in FIG. 55A) to a second usage status (as shown in FIG. 55B),the display mode selection is updated. In response to the identifiedchange in the usage status, computing device 5514 updates the displaymode selection from a first display mode 5510 (as shown in FIG. 55A) toa second display mode 5511 (as shown in FIG. 55B). Additionally, whencomputing device 5514 identifies the usage status of the wearableextended reality appliance 5513 has changed from a second usage status(as shown in FIG. 55B) to a first usage status (as shown in FIG. 55A),the display mode selection is updated. In response to the identifiedchange in the usage status, computing device 5514 updates the displaymode selection from a second display mode 5511 (as shown in FIG. 55B) toa first display mode 5510 (as shown in FIG. 55A).

Some embodiments involve automatically revising the presentation of theplurality of digital objects in response to an identified change inusage status and/or in response to an updated display mode selection ofthe wearable extended reality appliance. In one embodiment, when achange from the first usage status to the second usage status isidentified, in response to the change in the usage status, the at leastone processor may perform operations for automatically revising thepresentation of the plurality of digital objects to be consistent withthe second display mode. In another embodiment, when the display mode isupdated from the second display mode to the first display mode, inresponse to the updated display mode selection, the at least oneprocessor may perform operations for automatically revising thepresentation of the plurality of digital objects to be consistent withthe first display mode.

As used herein, “revising the presentation of the plurality of digitalobjects” may relate to a change in the display of the plurality ofdigital objects to the user of the physical display and/or the wearableextended reality appliance. In some embodiments, the presentation of theplurality of digital objects may be revised in response to the updateddisplay mode. As used herein, the term “automatically” may refer to achange, in real time or near real time, to the presentation of theplurality of digital objects in response to the identified change in theusage status, and/or a resulting change to the display mode in view ofthe identified change in the usage status, that is made without humanintervention. In some embodiments, when the presentation of theplurality of digital objects is revised to be consistent with the seconddisplay mode, automatically revising the presentation of the pluralityof digital objects may include causing a first digital object from theplurality of digital objects to disappear from the physical display,causing the first digital object to be presented via the wearableextended reality appliance, and/or causing an additional digital objectexcluded from the plurality of digital objects to be presented via thewearable extended reality appliance. In some embodiments, when thepresentation of the plurality of digital objects is revised to beconsistent with the first display mode, automatically revising thepresentation of the plurality of digital objects may include causing afirst digital object and a second digital object previously presentedvia the wearable extended reality appliance to reappear on the physicaldisplay, and/or causing a third digital object previously presented viathe wearable extended reality appliance to disappear.

By way of a non-limiting example, referring to FIGS. 55A and 55B, thecomputing device 5514 is configured to determine whether the usagestatus of the wearable extended reality appliance 5513 has changed froma first usage status (as shown in FIG. 55A) to a second usage status (asshown in FIG. 55B). When the computing device 5514 identifies a changefrom the first usage status to the second usage status (e.g., user 5512puts on wearable extended reality appliance 5513 and wirelessly connectsthe wearable extended reality appliance 5513 to the computing device5514), computing device 5514 automatically revises the presentation ofthe plurality of digital objects to be consistent with the seconddisplay mode 5511 (as shown in FIG. 55B).

When the presentation of the plurality of digital objects is revised tobe consistent with the second display mode 5511, the automaticallyrevised presentation of the plurality of digital objects includes afirst digital object (real digital object 5519A in FIG. 55A) from theplurality of digital objects to disappear from the physical display 5515and causing the first digital object (virtual digital object 5520 inFIG. 55B) to be presented via the wearable extended reality appliance5513. Additionally, the automatically revised presentation of theplurality of digital objects includes an additional digital object(virtual digital object 5521) excluded from the plurality of digitalobjects shown in FIG. 55A to be presented via the wearable extendedreality appliance 5513 in FIG. 55B.

In another example, the computing device 5514 is configured to determinewhether the usage status of the wearable extended reality appliance 5513has changed from a second usage status (as shown in FIG. 55B) to a firstusage status (as shown in FIG. 55A). When the computing device 5514identifies a change from the second usage status to the first usagestatus (e.g., user 5512 takes off wearable extended reality appliance5513), computing device 5514 automatically revises the presentation ofthe plurality of digital objects to be consistent with the first displaymode 5510 (as shown in FIG. 55A).

When the presentation of the plurality of digital objects is revised tobe consistent with the first display mode 5510, the automaticallyrevised presentation of the plurality of digital objects includes afirst digital object (virtual digital object 5520 in FIG. 55B)previously presented via the wearable extended reality appliance 5513 toreappear on the physical display as real digital object 5519A in FIG.55A. Additionally, the automatically revised presentation of theplurality of digital objects excludes an additional digital object(virtual digital object 5521) previously presented via the wearableextended reality appliance 5513 from display via physical display 5515,such that virtual digital object 5521 is caused to disappear fromdisplay to user 5512.

Some embodiments involve a system for selectively controlling display ofdigital objects, the system comprising at least one processor programmedto: generate a plurality of digital objects for display in connectionwith use of a computing device operable in a first display mode and in asecond display mode, wherein in the first display mode, the plurality ofdigital objects are displayed via a physical display connected to thecomputing device, and in the second display mode, some of the pluralityof digital objects are displayed via the physical display and at leastone other of the plurality of digital objects is displayed via awearable extended reality appliance; determine a usage status of thewearable extended reality appliance; select a display mode based on theusage status of the wearable extended reality appliance; and in responseto the display mode selection, output for presentation the plurality ofdigital objects in a manner consistent with the selected display mode.

FIG. 57 illustrates a flowchart of an example process 5700 forselectively controlling a display of digital objects, consistent withsome embodiments of the present disclosure. In some embodiments, process5700 may be performed by at least one processor (e.g., one or more ofserver 210 of FIG. 2 , mobile communications device 206, processingdevice 360 of FIG. 3 , processing device 460 of FIG. 4 , processingdevice 560 of FIG. 5 ) to perform operations or functions describedherein. In some embodiments, some aspects of process 5700 may beimplemented as software (e.g., program codes or instructions) that arestored in a memory (e.g., any of memory devices 212, 311, 411, or 511,or a memory of mobile device 206) or a non-transitory computer readablemedium. In some embodiments, some aspects of process 5700 may beimplemented as hardware (e.g., a specific-purpose circuit). In someembodiments, process 5700 may be implemented as a combination ofsoftware and hardware.

Referring to FIG. 57 , process 5700 may include a step 5710 ofgenerating a plurality of digital objects for display in connection withuse of a computing device operable in a first display mode and in asecond display mode, wherein in the first display mode, the plurality ofdigital objects are displayed via a physical display connected to thecomputing device, and in the second display mode, some of the pluralityof digital objects are displayed via the physical display, and at leastone other of the plurality of digital objects is displayed via awearable extended reality appliance.

Process 5700 may include a step 5712 of determining a usage status ofthe wearable extended reality appliance. In some embodiments, the usagestatus of the wearable extended reality appliance may be determinedafter the plurality of digital objects are generated for display inconnection with use of a computing device by step 5710. By way ofexample, FIG. 58 illustrates one non-limiting example of a process 5800for determining a usage status of a wearable extended reality appliance.FIG. 58 is an exemplary representation of just one embodiment, and it isto be understood that some illustrated features might be omitted, andothers added within the scope of this disclosure.

Process 5800 may include a step 5810 of initiating determination of ausage status (e.g., a first usage status or a second usage status) ofthe wearable extended reality appliance. A processor (e.g., one or moreof server 210 of FIG. 2 , mobile communications device 206, processingdevice 360 of FIG. 3 , processing device 460 of FIG. 4 , processingdevice 560 of FIG. 5 ) may determine the usage status, for example,using data indicating when the wearable extended reality appliance isactive, when a communication channel is established, or when thewearable extended reality appliance is worn by the user, consistent withsome embodiments of the present disclosure.

Process 5800 may include a step 5812 of determining whether the wearableextended reality appliance is active, as described above. In oneexample, when the at least one processor determines that the wearableextended reality appliance is not active (e.g., the wearable extendedreality appliance is off), the at least one processor may determine atstep 5816 that the usage status of the wearable extended realityappliance is a first usage status. Alternatively, when the at least oneprocessor determines that the wearable extended reality appliance isactive (e.g., the wearable extended reality appliance is on), the atleast one processor may proceed to step 5813. In another embodiment,when the at least one processor determines that the wearable extendedreality appliance is active, the at least one processor may determinethe usage status of the wearable extended reality appliance is a secondusage status.

Process 5800 may include a step 5813 of determining whether acommunication channel is established between the computing device andthe wearable extended reality appliance, as described above. In oneexample, when the at least one processor determines that a communicationchannel is not established between the computing device and the wearableextended reality appliance (e.g., the wearable extended realityappliance is disconnected from the computing device), the at least oneprocessor may determine at step 5816 the usage status of the wearableextended reality appliance is the first usage status. Alternatively,when the at least one processor determines that a communication channelis established between the computing device and the wearable extendedreality appliance (e.g., the wearable extended reality appliance isconnected to the computing device), the at least one processor mayproceed to step S814. In another embodiment, when the at least oneprocessor determines that a communication channel is established betweenthe computing device and the wearable extended reality appliance, the atleast one processor may determine the usage status of the wearableextended reality appliance is the second usage status.

Process 5800 may include a step 5814 of determining whether the wearableextended reality appliance is worn by the user, as described above. Inone example, when the at least one processor determines that thewearable extended reality appliance is not worn by the user (e.g., aproximity sensor detects the wearable extended reality appliance is notin proximity to the computing device and/or an image sensor detects thewearable extended reality appliance is not properly worn), the at leastone processor may determine at step 5816 the usage status of thewearable extended reality appliance is the first usage status.Alternatively, when the at least one processor determines that thewearable extended reality appliance is worn by the user (e.g., aproximity sensor detects the wearable extended reality appliance is inproximity to the computing device and/or an image sensor detects thewearable extended reality appliance is properly worn), the at least oneprocessor may determine at step 5818 the usage status of the wearableextended reality appliance is a second usage status.

Referring back to FIG. 57 , process 5700 may include a step 5714 ofselecting a display mode based on the usage status of the wearableextended reality appliance. In some embodiments, the display mode may beselected after a usage status of the wearable extended reality applianceis determined by step 5712. By way of example, FIG. 59 illustrates onenon-limiting example of a process 5900 for selecting a display mode of awearable extended reality appliance based on a usage status of thewearable extended reality appliance. FIG. 59 is an exemplaryrepresentation of just one embodiment, and it is to be understood thatsome illustrated features might be omitted, and others added within thescope of this disclosure.

Process 5900 may include a step 5910 of initiating determination of adisplay mode selection based on a determined usage status of thewearable extended reality appliance, consistent with some embodiments ofthe present disclosure. A processor (e.g., one or more of server 210 ofFIG. 2 , mobile communications device 206, processing device 360 of FIG.3 , processing device 460 of FIG. 4 , processing device 560 of FIG. 5 )may determine the display mode (e.g., a first display mode or seconddisplay mode) based on a determination of whether the usage status ofthe wearable extended reality appliance is a first usage status or asecond usage status.

Process 5900 may include a step 5912 of determining whether the usagestatus of the wearable extended reality appliance is a first usagestatus, consistent with some embodiments of the present disclosure. Inone example, when the at least one processor determines that thewearable extended reality appliance is in the first usage status, the atleast one processor may select at step 5914 the first display mode ofthe computing device. When the at least one processor determines thatthe wearable extended reality appliance is not in the first usagestatus, the at least one processor may proceed to step 5916.

Process 5900 may include a step 5916 of determining whether the usagestatus of the wearable extended reality appliance is a second usagestatus, consistent with some embodiments of the present disclosure. Inone example, when the at least one processor determines that thewearable extended reality appliance is in the second usage status, theat least one processor may select at step 5918 the second display modeof the computing device. In one embodiment, when the at least oneprocessor determines that the wearable extended reality appliance is notin the second usage status at step 5916, the at least one processor mayretry step 5910. In another embodiment, the at least one processor maydetermine whether the usage status of the wearable extended realityappliance is a second usage status prior to determining whether theusage status of the wearable extended reality appliance is a first usagestatus.

Referring back to FIG. 57 , process 5700U may include a step 5716 of, inresponse to the display mode selection, outputting for presentation theplurality of digital objects in a manner consistent with the selecteddisplay mode. In one embodiment, the plurality of digital objects may beoutput for presentation after a display mode of the wearable extendedreality appliance is selected by step 5714. By way of example, FIG. 59illustrates one non-limiting example of a process 5900 for selecting adisplay mode of a wearable extended reality appliance and outputting forpresentation the plurality of digital objects in a manner consistentwith the selected display mode, consistent with some embodiments of thepresent disclosure.

Process 5900 may include a step 5914 of selecting the first display modeof the computing device based on the first usage status of the wearableextended reality appliance, as described above. When the at least oneprocessor selects the first display mode at step 5914, the at least oneprocessor outputs for presentation the plurality of digital objects in amanner consistent with the first display mode at step 5915. Process 5900may include a step 5918, of selecting the second display mode of thecomputing device based on the second usage status of the wearableextended reality appliance, as described above. When the at least oneprocessor selects the second display mode at step 5918, the at least oneprocessor outputs for presentation the plurality of digital objects in amanner consistent with the second display mode at step 5919.

FIG. 60 is a flowchart illustrating an exemplary process 6000 fordetermining to display certain digital objects in the second displaymode via a wearable extended reality appliance when the wearableextended reality appliance is in a second usage status, consistent withsome embodiments of the present disclosure. FIG. 60 is an exemplaryrepresentation of just one embodiment, and it is to be understood thatsome illustrated features might be omitted, and others added within thescope of this disclosure.

With reference to step 6010 of FIG. 60 , instructions contained in anon-transitory computer-readable medium when executed by at least oneprocessor may cause the at least one processor to analyze input signalsand stored data and/or information when the wearable extended realityappliance is in the second usage status to determine which digitalobjects are to be displayed in the second display mode, consistent withsome embodiments of the present disclosure. For example, the at leastone processor may analyze user input signals captured by at least onesensor in communication with a computing device and/or wearable extendedreality appliance. In step 6012, the at least one processor may becaused to access database 6014 related to the display of a plurality ofdigital objects for presentation in the second display mode. While onlyone database 6014 is depicted herein for illustrative purposes, it is tobe understood that the referenced data and/or information shown thereinmay be contained in and/or across any number of databases.

In one example, the at least one processor may access data and/orinformation stored in database 6014 related to at least one user input6011, described above. Based on data and/or information related to theat least one user input 6011, the at least one processor may determinewhich of the plurality of digital objects to display via the wearableextended reality appliance in the second display mode at step 6016. Inanother example, the at least one processor may access data and/orinformation stored in database 6014 related to at least one past useraction 6013, described above. Based on data and/or information relatedto the at least one past user action 6013, the at least one processormay determine which of the plurality of digital objects to display viathe wearable extended reality appliance in the second display mode atstep 6016.

In another example, the at least one processor may access data and/orinformation related to at least one predefined rule 6015, describedabove. Based on data and/or information related to the at least onepredefined rule 6015, the at least one processor may determine which ofthe plurality of digital objects to display via the wearable extendedreality appliance in the second display mode at step 6016. In anotherexample, the at least one processor may access data and/or informationstored in database 6014 related to at least one type of input deviceconnected 6017, described above. Based on data and/or informationrelated to the at least one type of input device connected 6017, the atleast one processor may determine which of the plurality of digitalobjects to display via the wearable extended reality appliance in thesecond display mode at step 6016. In yet another example, the at leastone processor may access data and/or information stored in database 6014related to any combination of user input 6011, past user actions 6013,predefined rules 6015, and/or type of input device connected 6017 todetermine which of the plurality of digital objects are to be displayedvia the wearable extended reality appliance in the second display modeat step 6016.

FIG. 61 is a flowchart illustrating a process 6100 for identifying achange in a usage status of a wearable extended reality appliance andrevising the presentation of a plurality of digital objects, consistentwith some embodiments of the present disclosure in response to theidentified change in the usage status of the wearable extended realityappliance. FIG. 61 is an exemplary representation of just oneembodiment, and it is to be understood that some illustrated featuresmight be omitted, and others added within the scope of this disclosure.

With reference to step 6110 of FIG. 61 , instructions contained in anon-transitory computer-readable medium when executed by at least oneprocessor may cause the at least one processor to identify a change inthe usage status of the wearable extended reality appliance. In step6112, the at least one processor may be caused to determine whether theusage status of the wearable extended reality appliance has changed froma first usage status to a second usage status, as described above. Whenthe at least one processor identifies a change from the first usagestatus to the second usage status at step 6112, the at least oneprocessor may be caused to revise the presentation of the plurality ofdigital objects to be consistent with the second display mode at step6114.

When the at least one processor determines that the usage status has notchanged from the first usage status to the second usage status, the atleast one processor may proceed to step 6116. In step 6116, the at leastone processor may be caused to determine whether the usage status of thewearable extended reality appliance has changed from a second usagestatus to a first usage status, as described above. When the at leastone processor identifies a change from the second usage status to thefirst usage status at step 6116, the at least one processor may becaused to revise the presentation of the plurality of digital objects tobe consistent with the first display mode at step 6118. In anotherembodiment, the at least one processor may be caused to make thedetermination at step 6116 at the same time as and/or before thedetermination at step 6112.

Implementation of the method and system of the present disclosure mayinvolve performing or completing certain selected tasks or stepsmanually, automatically, or a combination thereof. Moreover, accordingto actual instrumentation and equipment of preferred embodiments of themethod and system of the present disclosure, several selected steps maybe implemented by hardware (HW) or by software (SW) on any operatingsystem of any firmware, or by a combination thereof. For example, ashardware, selected steps of the disclosure could be implemented as achip or a circuit. As software or algorithm, selected steps of thedisclosure could be implemented as a plurality of software instructionsbeing executed by a computer using any suitable operating system. In anycase, selected steps of the method and system of the disclosure could bedescribed as being performed by a data processor, such as a computingdevice for executing a plurality of instructions.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back end, middleware, orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (“LAN”), a wide area network (“WAN”), and theInternet. The computing system can include clients and servers. A clientand server are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While certain features of the described implementations have beenillustrated as described herein, many modifications, substitutions,changes and equivalents will now occur to those skilled in the art. Itis, therefore, to be understood that the appended claims are intended tocover all such modifications and changes as fall within the scope of theimplementations. It should be understood that they have been presentedby way of example only, not limitation, and various changes in form anddetails may be made. Any portion of the apparatus and/or methodsdescribed herein may be combined in any combination, except mutuallyexclusive combinations. The implementations described herein can includevarious combinations and/or sub-combinations of the functions,components and/or features of the different implementations described.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations of theembodiments will be apparent from consideration of the specification andpractice of the disclosed embodiments. For example, the describedimplementations include hardware and software, but systems and methodsconsistent with the present disclosure may be implemented as hardwarealone.

It is appreciated that the above-described embodiments can beimplemented by hardware, or software (program codes), or a combinationof hardware and software. If implemented by software, it can be storedin the above-described computer-readable media. The software, whenexecuted by the processor can perform the disclosed methods. Thecomputing units and other functional units described in the presentdisclosure can be implemented by hardware, or software, or a combinationof hardware and software. One of ordinary skill in the art will alsounderstand that multiple ones of the above-described modules/units canbe combined as one module or unit, and each of the above-describedmodules/units can be further divided into a plurality of sub-modules orsub-units.

The block diagrams in the figures illustrate the architecture,functionality, and operation of possible implementations of systems,methods, and computer hardware or software products according to variousexample embodiments of the present disclosure. In this regard, eachblock in a flowchart or block diagram may represent a module, segment,or portion of code, which includes one or more executable instructionsfor implementing the specified logical functions. It should beunderstood that in some alternative implementations, functions indicatedin a block may occur out of order noted in the figures. For example, twoblocks shown in succession may be executed or implemented substantiallyconcurrently, or two blocks may sometimes be executed in reverse order,depending upon the functionality involved. Some blocks may also beomitted. It should also be understood that each block of the blockdiagrams, and combination of the blocks, may be implemented by specialpurpose hardware-based systems that perform the specified functions oracts, or by combinations of special purpose hardware and computerinstructions.

In the foregoing specification, embodiments have been described withreference to numerous specific details that can vary from implementationto implementation. Certain adaptations and modifications of thedescribed embodiments can be made. Other embodiments can be apparent tothose skilled in the art from consideration of the specification andpractice of the invention disclosed herein. It is intended that thespecification and examples be considered as example only, with a truescope and spirit of the invention being indicated by the followingclaims. It is also intended that the sequence of steps shown in figuresare only for illustrative purposes and are not intended to be limited toany particular sequence of steps. As such, those skilled in the art canappreciate that these steps can be performed in a different order whileimplementing the same method.

It will be appreciated that the embodiments of the present disclosureare not limited to the exact construction that has been described aboveand illustrated in the accompanying drawings, and that variousmodifications and changes may be made without departing from the scopethereof. And other embodiments will be apparent to those skilled in theart from consideration of the specification and practice of thedisclosed embodiments disclosed herein. It is intended that thespecification and examples be considered as exemplary only, with a truescope and spirit of the disclosed embodiments being indicated by thefollowing claims.

Moreover, while illustrative embodiments have been described herein, thescope includes any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations or alterations based on the presentdisclosure. The elements in the claims are to be interpreted broadlybased on the language employed in the claims and not limited to examplesdescribed in the present specification or during the prosecution of theapplication. These examples are to be construed as non-exclusive.Further, the steps of the disclosed methods can be modified in anymanner, including by reordering steps or inserting or deleting steps. Itis intended, therefore, that the specification and examples beconsidered as exemplary only, with a true scope and spirit beingindicated by the following claims and their full scope of equivalents.

1-80. (canceled)
 81. A non-transitory computer readable mediumcontaining instructions for causing at least one processor to performoperations to enable collaboration between physical writers and virtualwriters, the operations comprising: receiving image data representing ahand of a first physical writer holding a physical marking implement andengaging with a physical surface to create tangible markings, whereinthe image data is received from an image sensor associated with awearable extended reality appliance worn by the first physical writer;transmitting information based on the image data to at least onecomputing device associated with at least one second virtual writer, tothereby enable the at least one second virtual writer to view thetangible markings created by the first physical writer; receiving fromthe at least one computing device annotation data representingadditional markings in relative locations with respect to the tangiblemarkings created by the first physical writer; and in response toreceiving the annotation data, causing the wearable extended realityappliance to overlay the physical surface with virtual markings in therelative locations.
 82. The non-transitory computer readable medium ofclaim 81, wherein the physical surface is a compilation of pages. 83.The non-transitory computer readable medium of claim 82, wherein thereceived annotation data represents first virtual markings associatedwith a first page of the compilation and second virtual markingsassociated with a second page of the compilation.
 84. The non-transitorycomputer readable medium of claim 83, wherein the compilation is anotebook, and the operations further include analyzing the image data todetermine that the notebook is opened to the first page, and causing thewearable extended reality appliance to overlay the first virtualmarkings on the first page of the notebook and exclude overlaying thesecond virtual markings on the first page of the notebook.
 85. Thenon-transitory computer readable medium of claim 84, wherein theoperations further include analyzing the image data to determine whenthe first page is turned; and causing the first virtual markings todisappear in response to the determination that the first page isturned.
 86. The non-transitory computer readable medium of claim 85,wherein the operations further include analyzing the image data todetermine when the first page is flipped back; and causing the firstvirtual markings to reappear in response to the determination that thefirst page is flipped back.
 87. The non-transitory computer readablemedium of claim 84, wherein the operations further include causing thewearable extended reality appliance to present a virtual representationof the second page with the second virtual markings away from thenotebook.
 88. The non-transitory computer readable medium of claim 81,wherein the annotation data includes cursor data associated with apointing device, and the operations further include analyzing the cursordata to determine the relative locations of the additional markings. 89.The non-transitory computer readable medium of claim 81, wherein theannotation data includes second image data representing a hand of anadditional physical writer holding a second physical marking implementand engaging with a second physical surface to create second tangiblemarkings, and the operations further include analyzing the second imagedata to determine the relative locations of the second tangiblemarkings.
 90. The non-transitory computer readable medium of claim 81,wherein the at least one second virtual writer includes a plurality ofvirtual writers, and the operations further include causing the wearableextended reality appliance to present in association with virtualmarkings made by a virtual writer of the plurality of virtual writers, avisual indicator indicative of an identity of the virtual writer. 91.The non-transitory computer readable medium of claim 90, wherein thevisual indicator includes at least one of an image of a virtual writer,a textual indicator of a virtual writer, or a symbol associated with avirtual writer.
 92. The non-transitory computer readable medium of claim81, wherein transmitting the image data to the at least one computingdevice associated with the at least one second virtual writer includestransmitting the image data to a group of computing devices associatedwith a group of virtual writers, and wherein the operations furtherinclude receiving input from the first physical writer designating oneor more virtual writers of the group of virtual writers forparticipation, and displaying virtual markings associated with the oneor more virtual writers of the group of virtual writers designated forparticipation while preventing display of virtual markings associatedwith others in the group of virtual writers not designated forparticipation.
 93. The non-transitory computer readable medium of claim81, wherein the at least one second virtual writer includes a pluralityof virtual writers, and the operations further include causing thewearable extended reality appliance to display a listing of individualspermitted to view the tangible markings of the first physical writer.94. The non-transitory computer readable medium of claim 81, wherein theoperations further include transmitting additional data to the at leastone computing device associated with the at least one second virtualwriter, to thereby enable the at least one second virtual writer to viewthe tangible markings with a visual indicator indicative of an identityof the first physical writer associated with the tangible markings. 95.The non-transitory computer readable medium of claim 81, wherein afteroverlaying the physical surface with the virtual markings, theoperations further include receiving input for causing a modification inthe virtual markings, and in response to receiving the input, causingthe wearable extended reality appliance to modify the virtual markings.96. The non-transitory computer readable medium of claim 95, wherein themodification includes at least one of a deletion of at least a portionthe virtual markings, changing a size of at least a portion the virtualmarkings, changing a color of at least a portion the virtual markings,or changing a location of at least a portion the virtual markings. 97.The non-transitory computer readable medium of claim 95, wherein theinput includes additional image data received from the image sensor, andthe operations further include analyzing the additional image data toidentify a gesture indicting the modification in the virtual markings.98. The non-transitory computer readable medium of claim 81, wherein theoperations further include analyzing the image data to remove at leastone of the hand or the physical marking implement from the image data tothereby create modified image data, and using the modified image data toenable the at least one second virtual writer to view the tangiblemarkings created by the first physical writer without the at least oneof the hand or the physical marking implement.
 99. A system for enablingcollaboration between physical writers and virtual writers, the systemcomprising: at least one processor programmed to: receive image datarepresenting a hand of a first physical writer holding a physicalmarking implement and engaging with a physical surface to createtangible markings, wherein the image data is received from an imagesensor associated with a wearable extended reality appliance worn by thefirst physical writer; transmit information based on the image data toat least one computing device associated with at least one secondvirtual writer, to thereby enable the at least one second virtual writerto view the tangible markings created by the first physical writer;receive from the at least one computing device annotation datarepresenting additional markings in relative locations with respect tothe tangible markings created by the first physical writer; and inresponse to receiving the annotation data, cause the wearable extendedreality appliance to overlay the physical surface with virtual markingsin the relative locations.
 100. A method for enabling collaborationbetween physical writers and virtual writers, the method comprising:receiving image data representing a hand of a first physical writerholding a physical marking implement and engaging with a physicalsurface to create tangible markings, wherein the image data is receivedfrom an image sensor associated with a wearable extended realityappliance worn by the first physical writer; transmitting informationbased on the image data to at least one computing device associated withat least one second virtual writer, to thereby enable the at least onesecond virtual writer to view the tangible markings created by the firstphysical writer; receiving from the at least one computing deviceannotation data representing additional markings in relative locationswith respect to the tangible markings created by the first physicalwriter; and in response to receiving the annotation data, causing thewearable extended reality appliance to overlay the physical surface withvirtual markings in the relative locations. 101-160. (canceled)